Skip to content

First CPU implementation of HashedNets compression algorithm for DNNs #220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adi-sharma wants to merge 42 commits into from
Closed

First CPU implementation of HashedNets compression algorithm for DNNs #220

Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
818afb9
included xxhash.h
adi-sharma Jun 23, 2016
db6990f
added hashing initializations
adi-sharma Jun 24, 2016
efa2efb
added weightBucket
adi-sharma Jun 24, 2016
c821a57
edited applyWeights() without drop-out as an example
adi-sharma Jun 28, 2016
b6328a7
Updated all applyWeights() with HashedNets
adi-sharma Jul 1, 2016
a58ac1d
Update NeuralNet.h
adi-sharma Jul 1, 2016
c81f248
updated update() and Steepest Gradient Descent
adi-sharma Jul 1, 2016
96c8f42
Updated SGD operator() with HashedNets
adi-sharma Jul 1, 2016
0250775
updated weightDecay()
adi-sharma Jul 1, 2016
7c62f3b
updated weightDecay() with HashedNets
adi-sharma Jul 1, 2016
fe93bb2
Edited dropOutWeightFactor()
adi-sharma Jul 13, 2016
13d151a
Edited dropOutWeightFactor()
adi-sharma Jul 13, 2016
725bba3
Updated with HashedNets
adi-sharma Jul 18, 2016
4ec8917
Updated train() with HashedNets
adi-sharma Jul 18, 2016
b1caf73
Updated with complete HashedNets
adi-sharma Aug 23, 2016
c696958
Updated with complete HashedNets
adi-sharma Aug 23, 2016
8d6b2d4
Updated with Complete HashedNets
adi-sharma Aug 23, 2016
e277c73
Update MethodDNN.h
adi-sharma Aug 23, 2016
a820db6
Update NeuralNet.h
adi-sharma Aug 23, 2016
919a319
Updated with HashedNets
adi-sharma Aug 23, 2016
4d9e46c
Error corrections
adi-sharma Aug 24, 2016
3d7eec3
Error corrections NeuralNet.icc
adi-sharma Aug 24, 2016
d202ae1
Error corrections MethodDNN.cxx
adi-sharma Aug 24, 2016
52dc851
Successful compile
adi-sharma Aug 25, 2016
70484aa
Successful compile
adi-sharma Aug 25, 2016
6a0932f
Successful compile
adi-sharma Aug 25, 2016
4f5b919
Successful compile
adi-sharma Aug 25, 2016
96b29df
Successful build NeuralNet.h
adi-sharma Aug 25, 2016
f301784
Successful build NeuralNet.icc
adi-sharma Aug 25, 2016
9f1dce5
Successful build NeuralNet.cxx
adi-sharma Aug 25, 2016
f5e6943
Update NeuralNet.icc
adi-sharma Aug 26, 2016
c4f8749
Update NeuralNet.h
adi-sharma Aug 26, 2016
9637a69
Update MethodDNN.cxx
adi-sharma Aug 26, 2016
f62fcf2
Update NeuralNet.cxx
adi-sharma Aug 26, 2016
e48d24f
Made some logical changes in HashedNets
adi-sharma Aug 27, 2016
2bf3295
Made some logical changes in HashedNets
adi-sharma Aug 27, 2016
e048347
Update NeuralNet.icc
adi-sharma Aug 28, 2016
24c19c2
Production version v1.0
adi-sharma Aug 28, 2016
3c7f7ac
Production Version v1.0
adi-sharma Aug 28, 2016
e6332d5
Production Version v1.0
adi-sharma Aug 28, 2016
5c33612
Production Version v1.0
adi-sharma Aug 28, 2016
6eed664
Production Version v1.0
adi-sharma Aug 28, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Production Version v1.0
  • Loading branch information
@adi-sharma
adi-sharma authored Aug 28, 2016
commit 5c3361292a061eec13ad347b29b21782db46085d
28 changes: 15 additions & 13 deletions tmva/tmva/src/MethodDNN.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -428,6 +428,8 @@ void TMVA::MethodDNN::ProcessOptions()
std::vector<double> dropConfig;
dropConfig = fetchValue (block, "DropConfig", dropConfig);
int dropRepetitions = fetchValue (block, "DropRepetitions", 3);
int bucketSize = fetchValue (block, "BucketSize", 8);
fBucketSize = bucketSize;

TMVA::DNN::EnumRegularization eRegularization = TMVA::DNN::EnumRegularization::NONE;
if (regularization == "L1")
Expand All @@ -451,7 +453,7 @@ void TMVA::MethodDNN::ProcessOptions()
std::shared_ptr<TMVA::DNN::ClassificationSettings> ptrSettings = make_shared <TMVA::DNN::ClassificationSettings> (
GetName (),
convergenceSteps, batchSize,
testRepetitions, factorWeightDecay,
testRepetitions, factorWeightDecay, bucketSize,
eRegularization, fScaleToNumEvents, TMVA::DNN::MinimizerType::fSteepest,
learningRate,
momentum, repetitions, multithreading);
Expand All @@ -463,7 +465,7 @@ void TMVA::MethodDNN::ProcessOptions()
std::shared_ptr<TMVA::DNN::Settings> ptrSettings = make_shared <TMVA::DNN::Settings> (
GetName (),
convergenceSteps, batchSize,
testRepetitions, factorWeightDecay,
testRepetitions, factorWeightDecay, bucketSize,
eRegularization, TMVA::DNN::MinimizerType::fSteepest,
learningRate,
momentum, repetitions, multithreading);
Expand All @@ -474,7 +476,7 @@ void TMVA::MethodDNN::ProcessOptions()
std::shared_ptr<TMVA::DNN::Settings> ptrSettings = make_shared <TMVA::DNN::Settings> (
GetName (),
convergenceSteps, batchSize,
testRepetitions, factorWeightDecay,
testRepetitions, factorWeightDecay, bucketSize,
eRegularization, TMVA::DNN::MinimizerType::fSteepest,
learningRate,
momentum, repetitions, multithreading);
Expand Down Expand Up @@ -602,7 +604,7 @@ void TMVA::MethodDNN::Train()

// initialize weights
fNet.initializeWeights (fWeightInitializationStrategy,
std::back_inserter (fWeightBucket), layerWeightNumber);
std::back_inserter (fWeightBucket), layerWeightNumber, fBucketSize);
}


Expand Down Expand Up @@ -659,7 +661,7 @@ Double_t TMVA::MethodDNN::GetMvaValue( Double_t* /*errLower*/, Double_t* /*errUp
const std::vector<Float_t>& inputValues = GetEvent ()->GetValues ();
std::vector<double> input (inputValues.begin (), inputValues.end ());
input.push_back (1.0); // bias node
std::vector<double> output = fNet.compute (input, fWeightBucket);
std::vector<double> output = fNet.compute (input, fWeightBucket, fBucketSize);
if (output.empty ())
return 0.0;

Expand All @@ -680,7 +682,7 @@ const std::vector<Float_t> &TMVA::MethodDNN::GetRegressionValues()
const std::vector<Float_t>& inputValues = ev->GetValues ();
std::vector<double> input (inputValues.begin (), inputValues.end ());
input.push_back (1.0); // bias node
std::vector<double> output = fNet.compute (input, fWeightBucket);
std::vector<double> output = fNet.compute (input, fWeightBucket, fBucketSize);

if (fRegressionReturnVal == NULL) fRegressionReturnVal = new std::vector<Float_t>();
fRegressionReturnVal->clear();
Expand Down Expand Up @@ -724,7 +726,7 @@ const std::vector<Float_t> &TMVA::MethodDNN::GetMulticlassValues()
const std::vector<Float_t>& inputValues = GetEvent ()->GetValues ();
std::vector<double> input (inputValues.begin (), inputValues.end ());
input.push_back (1.0); // bias node
std::vector<double> output = fNet.compute (input, fWeightBucket);
std::vector<double> output = fNet.compute (input, fWeightBucket, fBucketSize);

// check the output of the network

Expand Down Expand Up @@ -1100,9 +1102,9 @@ void TMVA::MethodDNN::checkGradients ()
fNet.addLayer (DNN::Layer (outputSize, DNN::EnumFunction::LINEAR, DNN::ModeOutputValues::SIGMOID));
fNet.setErrorFunction (DNN::ModeErrorFunction::CROSSENTROPY);
// net.setErrorFunction (ModeErrorFunction::SUMOFSQUARES);

const int BUCKET_SIZE = 8;
Copy link
Contributor

@omazapa omazapa Aug 29, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Aditya,

Why bucket size is constant value here?

Copy link
Author

@adi-sharma adi-sharma Aug 29, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Omar, that's because this is the checkGradients () function. As you can see above at lines 1100 - 03, this is a predefined Neural Net created just to see if the algorithm calculates gradients properly. This is independent of the actual training.

size_t numWeights = fNet.numWeights (inputSize);
std::vector<double> weightBucket (numWeights / TMVA::DNN::BUCKET_SIZE);
std::vector<double> weightBucket (numWeights / BUCKET_SIZE);
//weights.at (0) = 1000213.2;

std::vector<Pattern> pattern;
Expand All @@ -1122,7 +1124,7 @@ void TMVA::MethodDNN::checkGradients ()
}


DNN::Settings settings (TString ("checkGradients"), /*_convergenceSteps*/ 15, /*_batchSize*/ 1, /*_testRepetitions*/ 7, /*_factorWeightDecay*/ 0, /*regularization*/ TMVA::DNN::EnumRegularization::NONE);
DNN::Settings settings (TString ("checkGradients"), /*_convergenceSteps*/ 15, /*_batchSize*/ 1, /*_testRepetitions*/ 7, /*_factorWeightDecay*/ 0, /*_bucketSize*/ 8, /*regularization*/ TMVA::DNN::EnumRegularization::NONE);

size_t improvements = 0;
size_t worsenings = 0;
Expand All @@ -1131,19 +1133,19 @@ void TMVA::MethodDNN::checkGradients ()
for (size_t iTest = 0; iTest < 1000; ++iTest)
{
TMVA::DNN::uniformDouble (weightBucket, 0.7);
std::vector<double> gradientBucket (numWeights / TMVA::DNN::BUCKET_SIZE, 0);
std::vector<double> gradientBucket (numWeights / BUCKET_SIZE, 0);
DNN::Batch batch (begin (pattern), end (pattern));
DNN::DropContainer dropContainer;
std::tuple<DNN::Settings&, DNN::Batch&, DNN::DropContainer&> settingsAndBatch (settings, batch, dropContainer);
double E = fNet (settingsAndBatch, weightBucket, gradientBucket);
std::vector<double> changedWeightBucket;
changedWeightBucket.assign (weightBucket.begin (), weightBucket.end ());

int changeWeightPosition = TMVA::DNN::randomInt (numWeights / TMVA::DNN::BUCKET_SIZE);
int changeWeightPosition = TMVA::DNN::randomInt (numWeights / BUCKET_SIZE);
double dEdw = gradientBucket.at (changeWeightPosition);
while (dEdw == 0.0)
{
changeWeightPosition = TMVA::DNN::randomInt (numWeights / TMVA::DNN::BUCKET_SIZE);
changeWeightPosition = TMVA::DNN::randomInt (numWeights / BUCKET_SIZE);
dEdw = gradientBucket.at (changeWeightPosition);
}

Expand Down