Argthtjtr

I am trying to select a good model, say between Random Forest and XGB. For that, I do this:

• stratified KFold (from model_selection)


• cross_val_score (from model_selection)


• get mean cv score for Random Forest classifier.

For XGB, even though I could do same as above, I want to use xgb.cv and Dtrain matrices - as they claim that is much faster and efficient. So, I did this:

• xgtrain = xgb.Dtrain(X, label=y)


• xgb_cv_results is a dataframe...with appropriate metric (I used 'mlogloss' with 'objective': 'multi:softprob').

Now, since the two cv scoring are different, I am not in a position to compare the scores to select the best model. How can I get similar scoring methods in cross_val_score and xgb.cv ? In other words, is the common scoring mechanism such as 'accuracy' or 'neg_log_loss' in both the cv methods?

Here, the metric is very different from scoring parameter seen in cross_val_score. Here, I am dealing with multi-class classification. And, I need to see soft probabilities ('multi:softprob'). With this option, I cannot use 'auc' as my metric (there is a known bug in XGBoost - for multi-class classification, it does not do proper checking of X and y) and errors out.

Any ideas?