Btukfyl

I'm struggling to combine Sequential Feature Selector (from mlxtend) with a GridSearchCV (from sklearn).

My objective is to perform a forward feature selection for each set of parameters, to find which combination of parameters and features produces the best score.
The following code is based on example 8 of the user guide from mlxtend (see http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/#example-8-sequential-feature-selection-and-gridsearch)

X = data.values #dataframe with 48 features and 200 rows

y = diags #binary classification for each row



X_train, X_test, y_train, y_test = train_test_split(

    X, y, test_size=0.3, random_state=0)





svm = svm.SVC()



sfs = SFS(estimator = svm,

          k_features = (1,len(data.columns)),

          forward = True,

          floating=False,

          scoring = 'f1',

          cv = 5)



pipe = Pipeline([

    ('sfs', sfs),

    ('svm', svm)

])



param_grid = [

    {

     'sfs__estimator__C': [0.001, 0.01, 0.1, 1, 10, 100, 1000],

     'sfs__estimator__gamma': ['auto', 'scale', 0.001, 0.0001],

     'sfs__estimator__kernel': ['linear', 'rbf'],

     }

]





gs = GridSearchCV(estimator = pipe,

                  param_grid = param_grid,

                  scoring = 'f1',

                  n_jobs=1,

                  cv=5,

                  refit = True)



gs.fit(X_train, y_train)



print("Best parameters via GridSearch", gs.best_params_)



print("nBest features:n", gs.best_estimator_.steps[0][1].k_feature_idx_)

print("nBest score:n", gs.best_estimator_.steps[0][1].k_score_)

When executing this code I get the following:

Best parameters via GridSearch {'sfs__estimator__gamma': 'auto', 'sfs__estimator__kernel': 'linear', 'sfs__estimator__C': 0.001}



Best features:

 (16, 39)



Best score:

 0.31333333333333335

I doubted these were the best results as I did actually get better results when setting a higher minimum number of features.

I noticed that changing the order of the grid of parameters modified the results. When using:

param_grid = [

    {

     # Notice the change of order for C, 0.01 is now first

     'sfs__estimator__C': [0.01, 0.001, 0.1, 1, 10, 100, 1000],

     'sfs__estimator__gamma': ['auto', 'scale', 0.001, 0.0001],

     'sfs__estimator__kernel': ['linear', 'rbf'],

     }

]

I got the following results:

Best parameters via GridSearch {'sfs__estimator__gamma': 'auto', 'sfs__estimator__kernel': 'linear', 'sfs__estimator__C': 0.01}



Best features:

 (16, 39)



Best score:

 0.4428571428571429

Best parameters seem to always return the first value of each parameter. I tried other combinations and the parameters where always the first ones, and the best features did't change.

Am I using GridSearchCV wrong? Or am I printing the wrong attributes?