🔼 🍸 ❗️ C＃での正則化を伴うエラー伝播アルゴリズム 🤦🏾 🦆 ◽️

こんにちは C＃で機械学習メソッドを実装するトピックを続けたいと思います。この記事では、直接分布ニューラルネットワークをトレーニングするためのエラーバック伝播アルゴリズムについて説明し、C＃での実装も紹介します。この実装の特徴は、アルゴリズムの実装が目的関数（ニューラルネットワークが最小化しようとしているもの）とニューロンの活性化関数の実装から抽象化されていることです。結果は一種のコンストラクターであり、これを使用してネットワークおよび学習アルゴリズムのさまざまなパラメーターを操作し、結果を確認および比較できます。すでに人工ニューラルネットワークとは何かに精通していることを前提としています（そうでない場合は、 Wikipediaまたはこれらの記事のいずれかを最初に学習することを強くお勧めします）。面白い？猫の下に登ります。

指定

まず、この記事で使用する表記法を検討します。基本概念を思い出すために、ニューロンとレイヤーの写真は提供しません。これはすべてウィキペディアとハブにありますそのため、すぐに戦闘に入ると、ニューロンの誘導されたローカルフィールド（または加算器）は次のようになります。

レイヤーnのj番目のニューロンの重みのベクトルと入力値のベクトルの線形結合の値です
b-ニューロンのシフトまたは変位; 入力ベクトルのゼロ値で常に1であることに同意する場合、その後、シフトはインデックスがゼロの重みとして指定でき、式は単純化できます
-層ニューロンの数n

ニューロン活性化関数、または加算値の伝達関数：

ネットワークの各ニューロンは、独自の活性化機能を持つことができます
最初のレイヤーを除くすべてのレイヤーでは、入力ベクトルは前のレイヤーの出力ベクトルになるため、

ニューロンからネットワーク自体に移動しましょう。ニューラルネットワークはモデルであり、パラメータがあります。学習アルゴリズムのタスクは、エラー関数の値を最小化するためにそのようなネットワークパラメータを選択することです。エラー関数はEで示されます。モデルのパラメーターはニューロンの重みです：

-レイヤーのi番目のニューロンに由来するレイヤーnのj番目のニューロンの重み（n-1）。

ギリシャ語これ

学習アルゴリズムのハイパーパラメータ-学習速度を示します。

重量の変化はdeltaで示されます。

勾配の方向は関数の値を増加させる方向を示していますが、最小化するには反対方向に移動する必要があります

したがって、新しいニューロンの重みは次のとおりです。

重みの変化に正規化を追加する（または必要とする）ことはまだ言及できます。正則化関数Rは、モデルパラメーターの関数です。この場合、これらはニューロンの重みです。したがって、新しいエラー関数はE + Rのようになり、重みを変更する式は次のように変換されます。

lambda-学習のハイパーパラメーター、正則化係数（学習速度に類似）
m-トレーニングサンプルのサイズ

一般的に、正則化の実装は学習アルゴリズムから抽象化することもできますが、学習アルゴリズムの現在の実装はとにかく最速ではないため、まだ実行していません。累積誤差、および別の正則化。別の理由は、ニューラルネットワークのトレーニングに使用される正則化の種類があまり多くないことです（たとえば、L1とL2のみを知っています）。この実装では、L2ノルムを使用します。これは学習アルゴリズムの不可欠な部分です。

エラー伝播アルゴリズム

まず、トレーニングモードに焦点を当てましょう。重みを変更するにはいくつかの方法があります。

または各トレーニング例の後（リアルタイムトレーニング、オンライントレーニング、batchSize = 1）
トレーニングサンプル全体の変更を蓄積してから、すべての重みを変更します（フルバッチ、batchSize = trainingSet.Length）。
または、一定数のトレーニングサンプル（ミニバッチ、batchSize = any_number <trainingSet.Length）を実行した後

オンライン学習の状況を考慮すると、簡単になります。だから、勢いはネットワーク入力に来ました

ネットワークが応答しました

ただし、 xに対する正しい応答は

。

エラー関数Eの偏導関数を考えてみましょう。

- ただし、エラー関数は加算器を介してのみ表現でき、加算器は重みに依存するため、このエントリを次のように書き換えることができます。
- したがって、重量による加算器の偏導関数が計算され、その後、重量による目的関数の偏導関数の式を書き換えることができます。最後に私たちは得る

詳細な説明は、最後のレイヤーと残りのレイヤーの2つのブランチに分かれています。

出力層

出力層については、すべてが単純です。エラー修正については、重みの1つに従って目的関数の導関数を計算し、デルタ値を計算するだけです。目的関数はニューロンの出力値または活性化関数の値のみに完全に依存し、活性化関数自体は加算器のみに依存することを考慮します

ここで、出力層の誤差を計算するには、ニューロンのターゲット関数または活性化関数が何であるかに関係なく、ポイントで偏導関数の値を計算する必要があることがわかります。

隠れ層

ただし、レイヤーが出力されない場合は、後続のすべてのレイヤーのエラー値を累積する必要があります。

- さらに、次のレイヤーの加算器は現在のレイヤーの出力のみに依存し、現在のレイヤーの出力は現在のレイヤーの加算器のみに依存することを考慮します
- 前の層の出力の加算器の導関数は、ニューロン間の重みのみを提供します。これにより、主な結論が書き直されます。

PS：これは度ではなく、レイヤーインデックスであることを示すために、上付き文字に角かっこを入れるのを忘れたことに気付きました。 どこにも学位はありませんが、これに留意してください。

私たちは何を持っています：

特定の活性化関数の導関数の計算はまさにあなたが必要とするものです
次の層の加算器の値による目的関数の偏導関数の計算。最後のレイヤーではなく、最後から最初への重みの変化を計算しているので、ここのすべても単純です。この値は前のステップですでに計算されています
- 次の層が最後の場合、現在の層のこの値を計算し、ネットワーク全体にソリューションを拡張します

実装

エラー機能

式を完成させ、実装に移り、エラー関数の概念から始めましょう。私はそれをメトリックの形で提示しました（実際、これは事実です）。 CalculatePartialDerivaitveByV2Indexメソッドは、v2のインデックス変数によって入力ベクトルの関数の偏導関数の値を計算します。

public interface IMetrics<T> { double Calculate(T[] v1, T[] v2); /// <summary> /// Calculate value of partial derivative by v2[v2Index] /// </summary> T CalculatePartialDerivaitveByV2Index(T[] v1, T[] v2, int v2Index); }

したがって、ネットワークの実際の出力から最終層の誤差関数の偏導関数の値を計算できます

。

たとえば、いくつかの実装を書きましょう。

ユークリッド距離の半分の最小化

そして、派生物は次のようになります。

 internal class HalfSquaredEuclidianDistance : IMetrics<T> { public override double Calculate(double[] v1, double[] v2) { double d = 0; for (int i = 0; i < v1.Length; i++) { d += (v1[i] - v2[i]) * (v1[i] - v2[i]); } return 0.5 * d; } public override double CalculatePartialDerivaitveByV2Index(double[] v1, double[] v2, int v2Index) { return v2[v2Index] - v1[v2Index]; } }

対数尤度の最小化

 internal class Loglikelihood : IMetrics<double> { public override double Calculate(double[] v1, double[] v2) { double d = 0; for (int i = 0; i < v1.Length; i++) { d += v1[i]*Math.Log(v2[i]) + (1 - v1[i])*Math.Log(1 - v2[i]); } return -d; } public override double CalculatePartialDerivaitveByV2Index(double[] v1, double[] v2, int v2Index) { return -(v1[v2Index]/v2[v2Index] - (1 - v1[v2Index])/(1 - v2[v2Index])); } }

ここでの主なことは、対数尤度が最初にマイナス記号で計算されることを忘れないことであり、微分もマイナスで行われます。私はチェックに焦点を合わせたり、ゼロによる除算やゼロの対数のケースを回避したりしません。

ニューロン活性化機能

同様に、ニューロンの活性化機能について説明します。

 public interface IFunction { double Compute(double x); double ComputeFirstDerivative(double x); }

そして例。

シグモイド

 internal class SigmoidFunction : IFunction { private double _alpha = 1; internal SigmoidFunction(double alpha) { _alpha = alpha; } public double Compute(double x) { double r = (1 / (1 + Math.Exp(-1 * _alpha * x))); //return r == 1f ? 0.9999999f : r; return r; } public double ComputeFirstDerivative(double x) { return _alpha * this.Compute(x) * (1 - this.Compute(x)); } }

双曲線正接

 internal class HyperbolicTangensFunction : IFunction { private double _alpha = 1; internal HyperbolicTangensFunction(double alpha) { _alpha = alpha; } public double Compute(double x) { return (Math.Tanh(_alpha * x)); } public double ComputeFirstDerivative(double x) { double t = Math.Tanh(_alpha*x); return _alpha*(1 - t*t); } }

ニューロン、レイヤー、ネットワーク

このセクションでは、ネットワークの主要な要素のプレゼンテーションを検討します。その実装は提供しません。彼女は明らかです。このアルゴリズムは、完全に接続された「レイヤード」ネットワークに対して提供されるため、それに応じてネットワークの実装を行う必要があります。

したがって、ニューロンは次のようになります。

 public interface INeuron { /// <summary> /// Weights of the neuron /// </summary> double[] Weights { get; } /// <summary> /// Offset/bias of neuron (default is 0) /// </summary> double Bias { get; set; } /// <summary> /// Compute NET of the neuron by input vector /// </summary> /// <param name="inputVector">Input vector (must be the same dimension as was set in SetDimension)</param> /// <returns>NET of neuron</returns> double NET(double[] inputVector); /// <summary> /// Compute state of neuron /// </summary> /// <param name="inputVector">Input vector (must be the same dimension as was set in SetDimension)</param> /// <returns>State of neuron</returns> double Activate(double[] inputVector); /// <summary> /// Last calculated state in Activate /// </summary> double LastState { get; set; } /// <summary> /// Last calculated NET in NET /// </summary> double LastNET { get; set; } IList<INeuron> Childs { get; } IList<INeuron> Parents { get; } IFunction ActivationFunction { get; set; } double dEdz { get; set; } }

なぜなら完全に接続された「階層化された」ネットワークを検討しているため、Childs and Parentsを実装することはできませんが、共通のアルゴリズムを作成する場合は、その必要があります。学習アルゴリズムにとって特に重要なフィールドを考慮してください。

LastNET-ニューロン加算器、最後に計算された値はここに保存されます
LastState-ニューロン出力、最後に計算された値はここに保存されます
dEdzは、前述のニューロンのdE / dzと同じであり、現在のニューロンがどの層にあるかに応じて計算されます。ニューロンの加算器に関する誤差関数の偏微分

ネットワーク層はよりシンプルに見えます：

 public interface ILayer { /// <summary> /// Compute output of the layer /// </summary> /// <param name="inputVector">Input vector</param> /// <returns>Output vector</returns> double[] Compute(double[] inputVector); /// <summary> /// Get last output of the layer /// </summary> double[] LastOutput { get; } /// <summary> /// Get neurons of the layer /// </summary> INeuron[] Neurons { get; } /// <summary> /// Get input dimension of neurons /// </summary> int InputDimension { get; } }

ネットワークビュー：

 public interface INeuralNetwork { /// <summary> /// Compute output vector by input vector /// </summary> /// <param name="inputVector">Input vector (double[])</param> /// <returns>Output vector (double[])</returns> double[] ComputeOutput(double[] inputVector); Stream Save(); /// <summary> /// Train network with given inputs and outputs /// </summary> /// <param name="inputs">Set of input vectors</param> /// <param name="outputs">Set if output vectors</param> void Train(IList<DataItem<double>> data); }

ただし、多層ニューラルネットワークを検討しているため、特別なビューが使用されます。

 public interface IMultilayerNeuralNetwork : INeuralNetwork { /// <summary> /// Get array of layers of network /// </summary> ILayer[] Layers { get; } }

学習アルゴリズム

学習アルゴリズムは戦略パターンによって実装されます：

 public interface ILearningStrategy<T> { /// <summary> /// Train neural network /// </summary> /// <param name="network">Neural network for training</param> /// <param name="inputs">Set of input vectors</param> /// <param name="outputs">Set of output vectors</param> void Train(T network, IList<DataItem<double>> data); }

理解を深めるために、この実装のコンテキストでニューラルネットワークの典型的なTrain関数を示します。

 public void Train(IList<DataItem<double>> data) { _learningStrategy.Train(this, data); }

入力形式

次の入力形式を使用します。

 public class DataItem<T> { private T[] _input = null; private T[] _output = null; public DataItem() { } public DataItem(T[] input, T[] output) { _input = input; _output = output; } public T[] Input { get { return _input; } set { _input = value; } } public T[] Output { get { return _output; } set { _output = value; } } }

前の部分のコードからわかるように、ニューラルネットワークは

 DataItem. 
      

        
        
        
      

     
      

        
        
        
      

        
      

        
        
        
      

          ,         ( ),      : 
      

        
        
        
      

     public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
      DataItem. 
      

        
        
        
      

     
      

        
        
        
      

    
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        
      

        
        
        
      

          ,         ( ),      : 
      

        
        
        
      

     public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
 DataItem. 
      

        
        
        
      

     
      

        
        
        
      

    
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
        
      

        
        
        
      

          ,         ( ),      : 
      

        
        
        
      

     public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } }

アルゴリズム

最後に、コンテキスト全体を表示して、ニューラルネットワーク学習アルゴリズムの実際の実装に進むことができます。

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). ( ) : if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); , , : do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); , , . , batch , . lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } , , , : //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); : "", ( , ) dE/dz //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } : "", ( , ) dE/dz, , //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz = network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } , ( ), : //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError = 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); , : } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); . , , , . . , , , , , . : : . , , , , . -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
      

        
        
        
      

     
      

        
        
        
      

          (    )   : 
      

        
        
        
      

     if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

          ,          ,    : 
      

        
        
        
      

     do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

       ,        ,        .  ,  batch  ,   . 
      

        
        
        
      

     lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

         ,     ,   ,   : 
      

        
        
        
      

     //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        : 
      

        
        
        
      

      "",           (  ,       )        dE/dz          
      

        
        
        
      

     //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

               : 
      

        
        
        
      

           "",           (  ,       )        dE/dz,        ,              
      

        
        
        
      

     //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

        ,               (  ),  : 
      

        
        
        
      

     //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

             ,    : 
      

        
        
        
      

     } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     
      

        
        
        
      

     
      

        
        
        
      

      
      

        
        
        
      

      .      ,        ,       ,        .                .              ,       , , ,  ,        . 
      

        
        
        
      

         : 
      

        
        
        
      

      
      

        
        
        
      

       : 
      

        
        
        
      

      . 
      

        
        
        
      

     
      

        
        
        
      

     ,    ,           ,             ,    .                -)

C＃での正則化を伴うエラー伝播アルゴリズム

指定