reproducible result for Maximum entropy (maxent) classifier
I am trying to update the baseline code of nltk.classify.rte_classify to add more features in order to improve the accuracy of the model. It uses MaxentClassifier. My problem is that every time I execute my code I get different accuracy results (mentioned after the code. ). Usually, for scikit-learn classifiers, we have parameter 'random_state' to get reproducible result. I want to do the same for MaxentClassifier in my case. I checked in their documentation but I could not find anything similar to random_state as we have for scikit classifier.
from nltk.classify.util import accuracy
import nltk.classify.rte_classify as classify
def rte_classifier(algorithm):
from nltk.corpus import rte as rte_corpus
train_set = rte_corpus.pairs(['rte1_dev.xml', 'rte2_dev.xml', 'rte3_dev.xml'])
test_set = rte_corpus.pairs(['rte1_test.xml'])
featurized_train_set = classify.rte_featurize(train_set)
featurized_test_set = classify.rte_featurize(test_set)
# Train the classifier
print('Training classifier...')
if algorithm in ['GIS', 'IIS']: # Use default GIS/IIS MaxEnt algorithm
clf = nltk.MaxentClassifier.train(featurized_train_set, algorithm)
else:
err_msg = str(
"RTEClassifier only supports these algorithms:n "
" 'GIS', 'IIS'.n")
raise Exception(err_msg)
print('Testing classifier...')
acc = accuracy(clf, featurized_test_set)
print('Accuracy: %6.4f' % acc)
return clf
rte_classifier('GIS')
- 1st time : Accuracy: 0.5929
- 2nd time : Accuracy: 0.5908
- 3rd time : Accuracy: 0.5854
- 4th time : Accuracy: 0.5913
The variation in accuracy for the test set may look smaller but in my own dataset with high number of features, the difference sometime goes up to 10% .
python nltk rte maxent
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
asked Nov 20 '18 at 12:04
Singh
445
add a comment |
I am trying to update the baseline code of nltk.classify.rte_classify to add more features in order to improve the accuracy of the model. It uses MaxentClassifier. My problem is that every time I execute my code I get different accuracy results (mentioned after the code. ). Usually, for scikit-learn classifiers, we have parameter 'random_state' to get reproducible result. I want to do the same for MaxentClassifier in my case. I checked in their documentation but I could not find anything similar to random_state as we have for scikit classifier.
from nltk.classify.util import accuracy
import nltk.classify.rte_classify as classify
def rte_classifier(algorithm):
from nltk.corpus import rte as rte_corpus
train_set = rte_corpus.pairs(['rte1_dev.xml', 'rte2_dev.xml', 'rte3_dev.xml'])
test_set = rte_corpus.pairs(['rte1_test.xml'])
featurized_train_set = classify.rte_featurize(train_set)
featurized_test_set = classify.rte_featurize(test_set)
# Train the classifier
print('Training classifier...')
if algorithm in ['GIS', 'IIS']: # Use default GIS/IIS MaxEnt algorithm
clf = nltk.MaxentClassifier.train(featurized_train_set, algorithm)
else:
err_msg = str(
"RTEClassifier only supports these algorithms:n "
" 'GIS', 'IIS'.n")
raise Exception(err_msg)
print('Testing classifier...')
acc = accuracy(clf, featurized_test_set)
print('Accuracy: %6.4f' % acc)
return clf
rte_classifier('GIS')
- 1st time : Accuracy: 0.5929
- 2nd time : Accuracy: 0.5908
- 3rd time : Accuracy: 0.5854
- 4th time : Accuracy: 0.5913
The variation in accuracy for the test set may look smaller but in my own dataset with high number of features, the difference sometime goes up to 10% .
python nltk rte maxent
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
asked Nov 20 '18 at 12:04
Singh
445
There are a couple of things that would help, but your needs may depend on the maxent algorithm you have selected, etc. Please include your code to show exactly how you are usingMaxentClassifier. Better yet, could you provide a toy but complete (runnable) example that is as small as possible, but shows non-deterministic behavior? E.g., something where the odds are 50-50 and the selected result is not always the same.
– alexis
Nov 20 '18 at 15:10
I have updated my question. I am using default MaxEnt algorithm .. i.e. GIS/ IIS.
– Singh
Nov 20 '18 at 17:03
Much better. Please also say explicitly, in your question, what "different results" you see on repeated runs: Different accuracy every time? Different classes assigned to a particular test pair? (Which one?)
– alexis
Nov 20 '18 at 22:19
add a comment |
I am trying to update the baseline code of nltk.classify.rte_classify to add more features in order to improve the accuracy of the model. It uses MaxentClassifier. My problem is that every time I execute my code I get different accuracy results (mentioned after the code. ). Usually, for scikit-learn classifiers, we have parameter 'random_state' to get reproducible result. I want to do the same for MaxentClassifier in my case. I checked in their documentation but I could not find anything similar to random_state as we have for scikit classifier.
from nltk.classify.util import accuracy
import nltk.classify.rte_classify as classify
def rte_classifier(algorithm):
from nltk.corpus import rte as rte_corpus
train_set = rte_corpus.pairs(['rte1_dev.xml', 'rte2_dev.xml', 'rte3_dev.xml'])
test_set = rte_corpus.pairs(['rte1_test.xml'])
featurized_train_set = classify.rte_featurize(train_set)
featurized_test_set = classify.rte_featurize(test_set)
# Train the classifier
print('Training classifier...')
if algorithm in ['GIS', 'IIS']: # Use default GIS/IIS MaxEnt algorithm
clf = nltk.MaxentClassifier.train(featurized_train_set, algorithm)
else:
err_msg = str(
"RTEClassifier only supports these algorithms:n "
" 'GIS', 'IIS'.n")
raise Exception(err_msg)
print('Testing classifier...')
acc = accuracy(clf, featurized_test_set)
print('Accuracy: %6.4f' % acc)
return clf
rte_classifier('GIS')
- 1st time : Accuracy: 0.5929
- 2nd time : Accuracy: 0.5908
- 3rd time : Accuracy: 0.5854
- 4th time : Accuracy: 0.5913
The variation in accuracy for the test set may look smaller but in my own dataset with high number of features, the difference sometime goes up to 10% .
python nltk rte maxent
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
asked Nov 20 '18 at 12:04
Singh
445
I am trying to update the baseline code of nltk.classify.rte_classify to add more features in order to improve the accuracy of the model. It uses MaxentClassifier. My problem is that every time I execute my code I get different accuracy results (mentioned after the code. ). Usually, for scikit-learn classifiers, we have parameter 'random_state' to get reproducible result. I want to do the same for MaxentClassifier in my case. I checked in their documentation but I could not find anything similar to random_state as we have for scikit classifier.
from nltk.classify.util import accuracy
import nltk.classify.rte_classify as classify
def rte_classifier(algorithm):
from nltk.corpus import rte as rte_corpus
train_set = rte_corpus.pairs(['rte1_dev.xml', 'rte2_dev.xml', 'rte3_dev.xml'])
test_set = rte_corpus.pairs(['rte1_test.xml'])
featurized_train_set = classify.rte_featurize(train_set)
featurized_test_set = classify.rte_featurize(test_set)
# Train the classifier
print('Training classifier...')
if algorithm in ['GIS', 'IIS']: # Use default GIS/IIS MaxEnt algorithm
clf = nltk.MaxentClassifier.train(featurized_train_set, algorithm)
else:
err_msg = str(
"RTEClassifier only supports these algorithms:n "
" 'GIS', 'IIS'.n")
raise Exception(err_msg)
print('Testing classifier...')
acc = accuracy(clf, featurized_test_set)
print('Accuracy: %6.4f' % acc)
return clf
rte_classifier('GIS')
- 1st time : Accuracy: 0.5929
- 2nd time : Accuracy: 0.5908
- 3rd time : Accuracy: 0.5854
- 4th time : Accuracy: 0.5913
The variation in accuracy for the test set may look smaller but in my own dataset with high number of features, the difference sometime goes up to 10% .
python nltk rte maxent
python nltk rte maxent
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
asked Nov 20 '18 at 12:04
Singh
445
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
asked Nov 20 '18 at 12:04
Singh
445
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
edited Nov 23 '18 at 7:48
Aqueous Carlos
289213
289213
asked Nov 20 '18 at 12:04
Singh
445
asked Nov 20 '18 at 12:04
Singh
445
asked Nov 20 '18 at 12:04
Singh
445
445
There are a couple of things that would help, but your needs may depend on the maxent algorithm you have selected, etc. Please include your code to show exactly how you are usingMaxentClassifier. Better yet, could you provide a toy but complete (runnable) example that is as small as possible, but shows non-deterministic behavior? E.g., something where the odds are 50-50 and the selected result is not always the same.
– alexis
Nov 20 '18 at 15:10
I have updated my question. I am using default MaxEnt algorithm .. i.e. GIS/ IIS.
– Singh
Nov 20 '18 at 17:03
Much better. Please also say explicitly, in your question, what "different results" you see on repeated runs: Different accuracy every time? Different classes assigned to a particular test pair? (Which one?)
– alexis
Nov 20 '18 at 22:19
add a comment |
There are a couple of things that would help, but your needs may depend on the maxent algorithm you have selected, etc. Please include your code to show exactly how you are usingMaxentClassifier. Better yet, could you provide a toy but complete (runnable) example that is as small as possible, but shows non-deterministic behavior? E.g., something where the odds are 50-50 and the selected result is not always the same.
– alexis
Nov 20 '18 at 15:10
I have updated my question. I am using default MaxEnt algorithm .. i.e. GIS/ IIS.
– Singh
Nov 20 '18 at 17:03
Much better. Please also say explicitly, in your question, what "different results" you see on repeated runs: Different accuracy every time? Different classes assigned to a particular test pair? (Which one?)
– alexis
Nov 20 '18 at 22:19
There are a couple of things that would help, but your needs may depend on the maxent algorithm you have selected, etc. Please include your code to show exactly how you are using
MaxentClassifier. Better yet, could you provide a toy but complete (runnable) example that is as small as possible, but shows non-deterministic behavior? E.g., something where the odds are 50-50 and the selected result is not always the same.– alexis
Nov 20 '18 at 15:10
There are a couple of things that would help, but your needs may depend on the maxent algorithm you have selected, etc. Please include your code to show exactly how you are using
MaxentClassifier. Better yet, could you provide a toy but complete (runnable) example that is as small as possible, but shows non-deterministic behavior? E.g., something where the odds are 50-50 and the selected result is not always the same.– alexis
Nov 20 '18 at 15:10
I have updated my question. I am using default MaxEnt algorithm .. i.e. GIS/ IIS.
– Singh
Nov 20 '18 at 17:03
I have updated my question. I am using default MaxEnt algorithm .. i.e. GIS/ IIS.
– Singh
Nov 20 '18 at 17:03
Much better. Please also say explicitly, in your question, what "different results" you see on repeated runs: Different accuracy every time? Different classes assigned to a particular test pair? (Which one?)
– alexis
Nov 20 '18 at 22:19
Much better. Please also say explicitly, in your question, what "different results" you see on repeated runs: Different accuracy every time? Different classes assigned to a particular test pair? (Which one?)
– alexis
Nov 20 '18 at 22:19
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392632%2freproducible-result-for-maximum-entropy-maxent-classifier%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53392632%2freproducible-result-for-maximum-entropy-maxent-classifier%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There are a couple of things that would help, but your needs may depend on the maxent algorithm you have selected, etc. Please include your code to show exactly how you are using
MaxentClassifier. Better yet, could you provide a toy but complete (runnable) example that is as small as possible, but shows non-deterministic behavior? E.g., something where the odds are 50-50 and the selected result is not always the same.– alexis
Nov 20 '18 at 15:10
I have updated my question. I am using default MaxEnt algorithm .. i.e. GIS/ IIS.
– Singh
Nov 20 '18 at 17:03
Much better. Please also say explicitly, in your question, what "different results" you see on repeated runs: Different accuracy every time? Different classes assigned to a particular test pair? (Which one?)
– alexis
Nov 20 '18 at 22:19