Boruta usage demo
Setup
[ ]:
# ! pip install seaborn xgboost scikit-learn eboruta
[2]:
import logging
import typing as t
import numpy as np
import pandas as pd
import seaborn as sns
import shap
from catboost import CatBoostClassifier
from sklearn.datasets import make_classification
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier
from eBoruta import eBoruta, TrialData, Features, Dataset, setup_logger
[3]:
np.random.seed(666)
[4]:
def plot_imp_history(df_history: pd.DataFrame):
sns.lineplot(x='Step', y='Importance', hue='Feature', data=df_history)
sns.lineplot(x='Step', y='Threshold', data=df_history, linestyle='--', linewidth=4)
Basic usage
Single objective, RandomForestClassifier, default params.
[5]:
x, y = make_classification(100, 10, n_informative=2)
boruta = eBoruta()
boruta.fit(x, y);
Increase verbosity
Turn on logging to get a glimpse on what’s going on
[6]:
LOGGER = setup_logger(stdout_level=logging.DEBUG, logger=logging.getLogger('eBoruta'))
[7]:
boruta = eBoruta(verbose=2)
boruta.fit(x, y);
2025-02-04 10:11:09,972 INFO [algorithm--fit]: Trial 1: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,031 INFO [algorithm--fit]: 40.0% (4) recorded as hits
2025-02-04 10:11:10,034 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,034 INFO [algorithm--_report_trial]: Total summary: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,035 INFO [algorithm--fit]: Trial 2: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,094 INFO [algorithm--fit]: 30.0% (3) recorded as hits
2025-02-04 10:11:10,097 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,098 INFO [algorithm--_report_trial]: Total summary: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,100 INFO [algorithm--fit]: Trial 3: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,189 INFO [algorithm--fit]: 30.0% (3) recorded as hits
2025-02-04 10:11:10,193 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,193 INFO [algorithm--_report_trial]: Total summary: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,195 INFO [algorithm--fit]: Trial 4: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,258 INFO [algorithm--fit]: 30.0% (3) recorded as hits
2025-02-04 10:11:10,261 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,262 INFO [algorithm--_report_trial]: Total summary: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,264 INFO [algorithm--fit]: Trial 5: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,325 INFO [algorithm--fit]: 30.0% (3) recorded as hits
2025-02-04 10:11:10,329 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,329 INFO [algorithm--_report_trial]: Total summary: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,330 INFO [algorithm--fit]: Trial 6: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,390 INFO [algorithm--fit]: 40.0% (4) recorded as hits
2025-02-04 10:11:10,393 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,394 INFO [algorithm--_report_trial]: Total summary: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,395 INFO [algorithm--fit]: Trial 7: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,454 INFO [algorithm--fit]: 30.0% (3) recorded as hits
2025-02-04 10:11:10,457 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,458 INFO [algorithm--_report_trial]: Total summary: {'accepted': 0, 'rejected': 0, 'tentative': 10}
2025-02-04 10:11:10,459 INFO [algorithm--fit]: Trial 8: sampled trial data with shapes x_train: (100, 20), y_train: (100,), x_test: (100, 20), y_test: (100,)
2025-02-04 10:11:10,526 INFO [algorithm--fit]: 30.0% (3) recorded as hits
2025-02-04 10:11:10,529 INFO [algorithm--_report_trial]: Out of 10: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,529 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,531 INFO [algorithm--fit]: Trial 9: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:10,602 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:10,604 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:10,604 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,606 INFO [algorithm--fit]: Trial 10: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:10,673 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:10,675 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:10,675 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,677 INFO [algorithm--fit]: Trial 11: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:10,744 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:10,746 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:10,747 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,748 INFO [algorithm--fit]: Trial 12: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:10,816 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:10,818 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:10,818 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,820 INFO [algorithm--fit]: Trial 13: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:10,887 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:10,889 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:10,890 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,891 INFO [algorithm--fit]: Trial 14: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:10,957 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:10,960 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:10,960 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:10,962 INFO [algorithm--fit]: Trial 15: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,032 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,034 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,034 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,036 INFO [algorithm--fit]: Trial 16: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,104 INFO [algorithm--fit]: 0.0% (0) recorded as hits
2025-02-04 10:11:11,106 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,106 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,108 INFO [algorithm--fit]: Trial 17: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,175 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,177 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,177 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,179 INFO [algorithm--fit]: Trial 18: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,245 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,247 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,248 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,249 INFO [algorithm--fit]: Trial 19: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,321 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,323 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,324 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,325 INFO [algorithm--fit]: Trial 20: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,391 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,394 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,394 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,396 INFO [algorithm--fit]: Trial 21: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,467 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,469 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,469 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,471 INFO [algorithm--fit]: Trial 22: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,536 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,538 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,539 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,540 INFO [algorithm--fit]: Trial 23: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,606 INFO [algorithm--fit]: 0.0% (0) recorded as hits
2025-02-04 10:11:11,608 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,608 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,610 INFO [algorithm--fit]: Trial 24: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,679 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,681 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,681 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,683 INFO [algorithm--fit]: Trial 25: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,752 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,755 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,755 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,756 INFO [algorithm--fit]: Trial 26: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,823 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,825 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,826 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,828 INFO [algorithm--fit]: Trial 27: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,896 INFO [algorithm--fit]: 0.0% (0) recorded as hits
2025-02-04 10:11:11,899 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,899 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,900 INFO [algorithm--fit]: Trial 28: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:11,965 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:11,967 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:11,967 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:11,969 INFO [algorithm--fit]: Trial 29: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:12,036 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:12,038 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:12,038 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:12,040 INFO [algorithm--fit]: Trial 30: sampled trial data with shapes x_train: (100, 6), y_train: (100,), x_test: (100, 6), y_test: (100,)
2025-02-04 10:11:12,106 INFO [algorithm--fit]: 100.0% (1) recorded as hits
2025-02-04 10:11:12,109 INFO [algorithm--_report_trial]: Out of 1: {'accepted': 0, 'rejected': 0, 'tentative': 1}
2025-02-04 10:11:12,109 INFO [algorithm--_report_trial]: Total summary: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:12,117 INFO [algorithm--report_features]: Reporting at max step 30. Feature counts: {'accepted': 3, 'rejected': 6, 'tentative': 1}
2025-02-04 10:11:12,118 INFO [algorithm--report_features]: Feature 1 was marked at step 8 and threshold 0.02 as Accepted, having 0.09 importance ({'min': 0.08, 'max': 0.12, 'median': 0.09, 'mean': 0.09, 'std': 0.01}) and total number of hits 8
2025-02-04 10:11:12,118 INFO [algorithm--report_features]: Feature 10 was marked at step 8 and threshold 0.02 as Rejected, having 0.0 importance ({'min': 0.0, 'max': 0.01, 'median': 0.01, 'mean': 0.01, 'std': 0.0}) and total number of hits 0
2025-02-04 10:11:12,119 INFO [algorithm--report_features]: Feature 2 was marked at step 8 and threshold 0.02 as Accepted, having 0.04 importance ({'min': 0.03, 'max': 0.05, 'median': 0.04, 'mean': 0.04, 'std': 0.0}) and total number of hits 8
2025-02-04 10:11:12,119 INFO [algorithm--report_features]: Feature 3 was marked at step 8 and threshold 0.02 as Rejected, having 0.01 importance ({'min': 0.0, 'max': 0.01, 'median': 0.01, 'mean': 0.01, 'std': 0.0}) and total number of hits 0
2025-02-04 10:11:12,120 INFO [algorithm--report_features]: Feature 4 was marked at step 30 and threshold 0.06 as Tentative, having 0.07 importance ({'min': 0.01, 'max': 0.11, 'median': 0.08, 'mean': 0.07, 'std': 0.03}) and total number of hits 21
2025-02-04 10:11:12,121 INFO [algorithm--report_features]: Feature 5 was marked at step 8 and threshold 0.02 as Rejected, having 0.01 importance ({'min': 0.01, 'max': 0.01, 'median': 0.01, 'mean': 0.01, 'std': 0.0}) and total number of hits 0
2025-02-04 10:11:12,121 INFO [algorithm--report_features]: Feature 6 was marked at step 8 and threshold 0.02 as Rejected, having 0.0 importance ({'min': 0.0, 'max': 0.01, 'median': 0.0, 'mean': 0.01, 'std': 0.0}) and total number of hits 0
2025-02-04 10:11:12,122 INFO [algorithm--report_features]: Feature 7 was marked at step 8 and threshold 0.02 as Accepted, having 0.21 importance ({'min': 0.17, 'max': 0.22, 'median': 0.2, 'mean': 0.2, 'std': 0.02}) and total number of hits 8
2025-02-04 10:11:12,123 INFO [algorithm--report_features]: Feature 8 was marked at step 8 and threshold 0.02 as Rejected, having 0.01 importance ({'min': 0.01, 'max': 0.02, 'median': 0.01, 'mean': 0.01, 'std': 0.0}) and total number of hits 0
2025-02-04 10:11:12,123 INFO [algorithm--report_features]: Feature 9 was marked at step 8 and threshold 0.02 as Rejected, having 0.01 importance ({'min': 0.01, 'max': 0.02, 'median': 0.01, 'mean': 0.01, 'std': 0.0}) and total number of hits 0
Access features and history
[8]:
features = boruta.features_
features.accepted, features.rejected, features.tentative
[8]:
(array(['1', '2', '7'], dtype=object),
array(['3', '5', '6', '8', '9', '10'], dtype=object),
array(['4'], dtype=object))
[9]:
df = features.history
print(df.shape)
df.head()
(300, 6)
[9]:
| Feature | Step | Importance | Hit | Decision | Threshold | |
|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 0.081904 | 1 | Tentative | 0.017866 |
| 1 | 1 | 2 | 0.120700 | 1 | Tentative | 0.027072 |
| 2 | 1 | 3 | 0.102623 | 1 | Tentative | 0.020077 |
| 3 | 1 | 4 | 0.088259 | 1 | Tentative | 0.022102 |
| 4 | 1 | 5 | 0.077580 | 1 | Tentative | 0.018511 |
Note that n_rows = n_steps * n_features. df.dropna() cleans the table giving access to the last step for a feature where it was used.b
[10]:
df.dropna().groupby('Feature').tail(1)
[10]:
| Feature | Step | Importance | Hit | Decision | Threshold | |
|---|---|---|---|---|---|---|
| 7 | 1 | 8 | 0.094969 | 1 | Accepted | 0.023125 |
| 37 | 2 | 8 | 0.039087 | 1 | Accepted | 0.023125 |
| 67 | 3 | 8 | 0.005328 | 0 | Rejected | 0.023125 |
| 119 | 4 | 30 | 0.073942 | 1 | Tentative | 0.061385 |
| 127 | 5 | 8 | 0.008686 | 0 | Rejected | 0.023125 |
| 157 | 6 | 8 | 0.004911 | 0 | Rejected | 0.023125 |
| 187 | 7 | 8 | 0.209556 | 1 | Accepted | 0.023125 |
| 217 | 8 | 8 | 0.011646 | 0 | Rejected | 0.023125 |
| 247 | 9 | 8 | 0.007699 | 0 | Rejected | 0.023125 |
| 277 | 10 | 8 | 0.004598 | 0 | Rejected | 0.023125 |
Query history to inspect the selection process
[11]:
df[df['Feature'] == '7']
[11]:
| Feature | Step | Importance | Hit | Decision | Threshold | |
|---|---|---|---|---|---|---|
| 180 | 7 | 1 | 0.196537 | 1 | Tentative | 0.017866 |
| 181 | 7 | 2 | 0.173083 | 1 | Tentative | 0.027072 |
| 182 | 7 | 3 | 0.176907 | 1 | Tentative | 0.020077 |
| 183 | 7 | 4 | 0.214924 | 1 | Tentative | 0.022102 |
| 184 | 7 | 5 | 0.219287 | 1 | Tentative | 0.018511 |
| 185 | 7 | 6 | 0.204183 | 1 | Tentative | 0.015110 |
| 186 | 7 | 7 | 0.183643 | 1 | Tentative | 0.025346 |
| 187 | 7 | 8 | 0.209556 | 1 | Accepted | 0.023125 |
| 188 | 7 | 9 | NaN | NaN | Accepted | 0.062966 |
| 189 | 7 | 10 | NaN | NaN | Accepted | 0.068240 |
| 190 | 7 | 11 | NaN | NaN | Accepted | 0.063476 |
| 191 | 7 | 12 | NaN | NaN | Accepted | 0.052037 |
| 192 | 7 | 13 | NaN | NaN | Accepted | 0.084228 |
| 193 | 7 | 14 | NaN | NaN | Accepted | 0.070804 |
| 194 | 7 | 15 | NaN | NaN | Accepted | 0.083188 |
| 195 | 7 | 16 | NaN | NaN | Accepted | 0.096808 |
| 196 | 7 | 17 | NaN | NaN | Accepted | 0.068633 |
| 197 | 7 | 18 | NaN | NaN | Accepted | 0.063534 |
| 198 | 7 | 19 | NaN | NaN | Accepted | 0.088086 |
| 199 | 7 | 20 | NaN | NaN | Accepted | 0.065355 |
| 200 | 7 | 21 | NaN | NaN | Accepted | 0.064586 |
| 201 | 7 | 22 | NaN | NaN | Accepted | 0.062369 |
| 202 | 7 | 23 | NaN | NaN | Accepted | 0.092352 |
| 203 | 7 | 24 | NaN | NaN | Accepted | 0.059420 |
| 204 | 7 | 25 | NaN | NaN | Accepted | 0.065167 |
| 205 | 7 | 26 | NaN | NaN | Accepted | 0.060183 |
| 206 | 7 | 27 | NaN | NaN | Accepted | 0.075320 |
| 207 | 7 | 28 | NaN | NaN | Accepted | 0.061957 |
| 208 | 7 | 29 | NaN | NaN | Accepted | 0.061686 |
| 209 | 7 | 30 | NaN | NaN | Accepted | 0.061385 |
One can use history to produce plots
[12]:
plot_imp_history(df)
Explore params
[13]:
?eBoruta
Lower percentile threshold
[14]:
boruta = eBoruta(percentile=70).fit(x, y)
plot_imp_history(boruta.features_.history)
Lower p-value
[15]:
boruta = eBoruta(pvalue=0.005).fit(x, y)
plot_imp_history(boruta.features_.history)
Apply rough fix
This won’t overwrite existing boruta.features_ but will return a new Features instance. In the latter, the history will remain unchanged, but the accepted, rejected, and tentative attributes will be modified accordingly.
[16]:
fs = boruta.rough_fix(n_last_trials=10)
[17]:
fs.accepted, fs.rejected, fs.tentative
[17]:
(array(['1', '2', '4', '7'], dtype=object),
array(['3', '5', '6', '8', '9', '10'], dtype=object),
array([], dtype=object))
Use test set
[18]:
boruta = eBoruta(test_size=0.3, test_stratify=True).fit(x, y)
Advanced usage
Different models
In principle, the model can be any callable defining a fit method – classifier or regressor – as long as the importance calculation is defined. Note that one can define the latter manually (see below).
For instance, we’ll use the XGBClassifier and CatBoostClassifier below.
XGBClassifier
[19]:
boruta = eBoruta().fit(
x, y, model_type=XGBClassifier,
model_init_kwargs=dict(n_estimators=20, verbosity=0)
)
plot_imp_history(boruta.features_.history)
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
ntree_limit is deprecated, use `iteration_range` or model slicing instead.
CatBoostClassifier
[20]:
# shap with `approximate` is not supported for catboost currently
boruta = eBoruta().fit(
x, y, model_type=CatBoostClassifier,
model_init_kwargs=dict(iterations=20, verbose=False)
)
plot_imp_history(boruta.features_.history)
Custom importance measure
Any callable accepting an estimator or estimator together with the TrialData object and returning a numpy array with shape (n_test_features, ) will work.
[21]:
def get_imp(estimator):
# equivalent to the builtin importance getter
return estimator.feature_importances_
boruta = eBoruta(importance_getter=get_imp)
boruta.fit(x, y)
plot_imp_history(boruta.features_.history)
[22]:
def get_permutation_imp(estimator: t.Any, trial_data: TrialData) -> np.ndarray:
imp = permutation_importance(
estimator, trial_data.x_test, trial_data.y_test,
scoring='accuracy', n_jobs=-1
)
return np.array(imp['importances_mean'])
# Let's also use a different estimator, just for the sake of it
boruta = eBoruta(
importance_getter=get_permutation_imp
).fit(
x, y,
model_type=ExtraTreesClassifier,
model_init_kwargs=dict(n_estimators=20)
)
plot_imp_history(boruta.features_.history)
Non-ensemble classifier with custom importance evaluation
[23]:
boruta = eBoruta(
importance_getter=get_permutation_imp
).fit(
x, y, model_type=LogisticRegression
)
plot_imp_history(boruta.features_.history)
Multiple objectives
Built-in approach is basically averaging importance of each feature per objective. One can define a different aggregation strategy via custom importance getter.
[24]:
y2 = np.array([[y_, y_] for y_ in y])
Using built-in shap importance evaluation
[25]:
boruta = eBoruta().fit(x, y2)
plot_imp_history(boruta.features_.history)
Using
feature_importances_attribute
[26]:
# Using shap importance
boruta = eBoruta(shap_tree=False).fit(x, y2)
plot_imp_history(boruta.features_.history)
Using custom importance evaluation
Use-case: different aggregation strategy for multiple objectives. Below we’ll use maximum of importances for a feature across objectives instead of the default mean.
[27]:
# Using custom aggregation
def get_imp(estimator, trial_data: TrialData):
# equivalent to the builtin importance getter
explainer = shap.explainers.Tree(estimator)
imp = explainer.shap_values(trial_data.x_test, approximate=False)
imp = np.max(np.vstack([np.abs(v).mean(0) for v in imp]), axis=0)
return imp
boruta = eBoruta(importance_getter=get_imp).fit(x, y)
plot_imp_history(boruta.features_.history)
Using Callbacks
It can be any callable (including classes with mutable state), accepting and returning (Estimator, Feature, Dataset, Trial). Check callbacks module for additional examples.
CatBoostClassifierwith categorical features
[28]:
def handle_catboost_categorical(
estimator: CatBoostClassifier, features: Features,
dataset: Dataset, trial_data: TrialData, **kwargs
):
params = estimator.get_params()
params['cat_features'] = [c for c in trial_data.x_test.columns if 'cat' in c]
estimator = estimator.__class__(**params)
return estimator, features, dataset, trial_data, kwargs
[30]:
x_cat = boruta.dataset_.x.copy()
x_cat['1_cat'] = pd.Series(x_cat['1'].round(0).astype(int).astype('category'))
boruta = eBoruta().fit(
x_cat, y, model_type=CatBoostClassifier,
model_init_kwargs=dict(iterations=20, verbose=False),
callbacks_trial_start=[handle_catboost_categorical],
)
plot_imp_history(boruta.features_.history)
CatBoostClassifierwith adjusted number of iterations
[31]:
class AdjustIterations:
def __init__(self, min_iterations: int = 5):
self.min_iterations = min_iterations
def __call__(self, estimator: CatBoostClassifier, features: Features,
dataset: Dataset, trial_data: TrialData, **kwargs):
num_features = trial_data.x_test.shape[1]
num_iterations = max([self.min_iterations, num_features // 2])
params = estimator.get_params()
params['iterations'] = num_iterations
estimator = estimator.__class__(**params)
print(
f'Set the number of iterations to '
f'{estimator.get_param("iterations")} '
f'(num_features={num_features})'
)
return estimator, features, dataset, trial_data, kwargs
[32]:
boruta = eBoruta().fit(
x, y, model_type=CatBoostClassifier,
model_init_kwags=dict(iterations=20, verbose=False),
callbacks_trial_start=[AdjustIterations()]
)
plot_imp_history(boruta.features_.history)
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4946210 total: 861us remaining: 7.75ms
1: learn: 0.3665493 total: 1.63ms remaining: 6.5ms
2: learn: 0.2895006 total: 2.19ms remaining: 5.1ms
3: learn: 0.2283721 total: 2.97ms remaining: 4.46ms
4: learn: 0.1926571 total: 3.67ms remaining: 3.67ms
5: learn: 0.1573209 total: 4.24ms remaining: 2.83ms
6: learn: 0.1396606 total: 4.8ms remaining: 2.06ms
7: learn: 0.1120921 total: 5.29ms remaining: 1.32ms
8: learn: 0.0994281 total: 5.72ms remaining: 635us
9: learn: 0.0869798 total: 6.11ms remaining: 0us
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4946210 total: 584us remaining: 5.26ms
1: learn: 0.3674051 total: 1.16ms remaining: 4.62ms
2: learn: 0.2900619 total: 1.55ms remaining: 3.62ms
3: learn: 0.2324166 total: 1.96ms remaining: 2.94ms
4: learn: 0.1957315 total: 2.37ms remaining: 2.37ms
5: learn: 0.1712470 total: 3.11ms remaining: 2.07ms
6: learn: 0.1411823 total: 3.69ms remaining: 1.58ms
7: learn: 0.1226734 total: 4.14ms remaining: 1.03ms
8: learn: 0.1113530 total: 4.75ms remaining: 527us
9: learn: 0.0963836 total: 5.3ms remaining: 0us
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4946210 total: 518us remaining: 4.67ms
1: learn: 0.3604945 total: 1.02ms remaining: 4.08ms
2: learn: 0.2996103 total: 1.48ms remaining: 3.44ms
3: learn: 0.2422803 total: 2ms remaining: 3.01ms
4: learn: 0.2026260 total: 2.54ms remaining: 2.54ms
5: learn: 0.1640899 total: 3.14ms remaining: 2.09ms
6: learn: 0.1397083 total: 3.57ms remaining: 1.53ms
7: learn: 0.1147725 total: 3.96ms remaining: 990us
8: learn: 0.1069255 total: 4.44ms remaining: 492us
9: learn: 0.0953707 total: 4.98ms remaining: 0us
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4754703 total: 920us remaining: 8.29ms
1: learn: 0.3327668 total: 1.77ms remaining: 7.1ms
2: learn: 0.2655481 total: 2.86ms remaining: 6.66ms
3: learn: 0.2117947 total: 4.28ms remaining: 6.41ms
4: learn: 0.1902543 total: 5.58ms remaining: 5.58ms
5: learn: 0.1606359 total: 7.07ms remaining: 4.72ms
6: learn: 0.1346939 total: 8.64ms remaining: 3.7ms
7: learn: 0.1082224 total: 10.2ms remaining: 2.56ms
8: learn: 0.1002422 total: 11.6ms remaining: 1.28ms
9: learn: 0.0904362 total: 12.9ms remaining: 0us
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4946210 total: 683us remaining: 6.15ms
1: learn: 0.3504649 total: 1.1ms remaining: 4.41ms
2: learn: 0.2712362 total: 1.52ms remaining: 3.56ms
3: learn: 0.2214000 total: 1.92ms remaining: 2.87ms
4: learn: 0.1910622 total: 2.36ms remaining: 2.36ms
5: learn: 0.1562268 total: 2.94ms remaining: 1.96ms
6: learn: 0.1324780 total: 3.54ms remaining: 1.52ms
7: learn: 0.1097883 total: 4.2ms remaining: 1.05ms
8: learn: 0.1000567 total: 5.72ms remaining: 635us
9: learn: 0.0885574 total: 6.53ms remaining: 0us
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4946210 total: 699us remaining: 6.3ms
1: learn: 0.3458185 total: 1.31ms remaining: 5.24ms
2: learn: 0.2831531 total: 1.98ms remaining: 4.61ms
3: learn: 0.2319511 total: 3.26ms remaining: 4.89ms
4: learn: 0.2001389 total: 4.45ms remaining: 4.45ms
5: learn: 0.1710612 total: 5.72ms remaining: 3.81ms
6: learn: 0.1442473 total: 7.07ms remaining: 3.03ms
7: learn: 0.1228968 total: 8.5ms remaining: 2.13ms
8: learn: 0.1078391 total: 9.84ms remaining: 1.09ms
9: learn: 0.1025566 total: 11.2ms remaining: 0us
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4946210 total: 1.5ms remaining: 13.5ms
1: learn: 0.3738821 total: 3.3ms remaining: 13.2ms
2: learn: 0.3020800 total: 6.25ms remaining: 14.6ms
3: learn: 0.2449950 total: 8.06ms remaining: 12.1ms
4: learn: 0.2103692 total: 9.53ms remaining: 9.53ms
5: learn: 0.1638249 total: 11ms remaining: 7.32ms
6: learn: 0.1395729 total: 12.3ms remaining: 5.28ms
7: learn: 0.1178425 total: 13.7ms remaining: 3.44ms
8: learn: 0.1105916 total: 15.2ms remaining: 1.69ms
9: learn: 0.0987842 total: 16.5ms remaining: 0us
Set the number of iterations to 10 (num_features=20)
Learning rate set to 0.262987
0: learn: 0.4946210 total: 714us remaining: 6.43ms
1: learn: 0.3654352 total: 1.28ms remaining: 5.12ms
2: learn: 0.2912461 total: 1.77ms remaining: 4.14ms
3: learn: 0.2483871 total: 2.25ms remaining: 3.38ms
4: learn: 0.2093708 total: 2.72ms remaining: 2.72ms
5: learn: 0.1647805 total: 3.17ms remaining: 2.11ms
6: learn: 0.1375144 total: 3.62ms remaining: 1.55ms
7: learn: 0.1131292 total: 4.12ms remaining: 1.03ms
8: learn: 0.1075389 total: 4.62ms remaining: 513us
9: learn: 0.0953702 total: 5.14ms remaining: 0us
Set the number of iterations to 5 (num_features=10)
Learning rate set to 0.496568
0: learn: 0.6309444 total: 1.01ms remaining: 4.06ms
1: learn: 0.5751024 total: 1.73ms remaining: 2.6ms
2: learn: 0.5380283 total: 2.35ms remaining: 1.57ms
3: learn: 0.5103902 total: 3.05ms remaining: 763us
4: learn: 0.4840500 total: 3.89ms remaining: 0us
Set the number of iterations to 5 (num_features=10)
Learning rate set to 0.496568
0: learn: 0.6508693 total: 1.09ms remaining: 4.34ms
1: learn: 0.6028759 total: 2.21ms remaining: 3.31ms
2: learn: 0.5772686 total: 3.03ms remaining: 2.02ms
3: learn: 0.5517780 total: 3.87ms remaining: 967us
4: learn: 0.5072810 total: 4.66ms remaining: 0us
Set the number of iterations to 5 (num_features=10)
Learning rate set to 0.496568
0: learn: 0.6431084 total: 437us remaining: 1.75ms
1: learn: 0.5962203 total: 673us remaining: 1.01ms
2: learn: 0.5683787 total: 906us remaining: 604us
3: learn: 0.5386821 total: 1.14ms remaining: 285us
4: learn: 0.5104961 total: 1.38ms remaining: 0us
Set the number of iterations to 5 (num_features=10)
Learning rate set to 0.496568
0: learn: 0.6669508 total: 554us remaining: 2.22ms
1: learn: 0.6193979 total: 970us remaining: 1.46ms
2: learn: 0.5828810 total: 1.41ms remaining: 938us
3: learn: 0.5486887 total: 2.11ms remaining: 528us
4: learn: 0.5240405 total: 2.94ms remaining: 0us
Set the number of iterations to 5 (num_features=9)
Learning rate set to 0.496568
0: learn: 0.6443430 total: 390us remaining: 1.56ms
1: learn: 0.5927087 total: 647us remaining: 970us
2: learn: 0.5675871 total: 982us remaining: 655us
3: learn: 0.5367640 total: 1.32ms remaining: 331us
4: learn: 0.5092877 total: 1.61ms remaining: 0us
Set the number of iterations to 5 (num_features=9)
Learning rate set to 0.496568
0: learn: 0.6490432 total: 1.1ms remaining: 4.4ms
1: learn: 0.6071164 total: 1.94ms remaining: 2.92ms
2: learn: 0.5840577 total: 2.71ms remaining: 1.81ms
3: learn: 0.5480852 total: 3.41ms remaining: 851us
4: learn: 0.5202422 total: 4.13ms remaining: 0us
Set the number of iterations to 5 (num_features=9)
Learning rate set to 0.496568
0: learn: 0.6579083 total: 475us remaining: 1.9ms
1: learn: 0.6087601 total: 849us remaining: 1.27ms
2: learn: 0.5790627 total: 1.18ms remaining: 786us
3: learn: 0.5519416 total: 1.51ms remaining: 377us
4: learn: 0.5130639 total: 1.96ms remaining: 0us
Set the number of iterations to 5 (num_features=9)
Learning rate set to 0.496568
0: learn: 0.6173739 total: 423us remaining: 1.69ms
1: learn: 0.5868192 total: 650us remaining: 975us
2: learn: 0.5657496 total: 874us remaining: 583us
3: learn: 0.5413902 total: 1.07ms remaining: 267us
4: learn: 0.5010316 total: 1.27ms remaining: 0us
Set the number of iterations to 5 (num_features=8)
Learning rate set to 0.496568
0: learn: 0.6245046 total: 621us remaining: 2.49ms
1: learn: 0.5897435 total: 1.11ms remaining: 1.67ms
2: learn: 0.5523369 total: 1.63ms remaining: 1.09ms
3: learn: 0.5192248 total: 4.05ms remaining: 1.01ms
4: learn: 0.4947232 total: 4.95ms remaining: 0us
Set the number of iterations to 5 (num_features=8)
Learning rate set to 0.496568
0: learn: 0.6361515 total: 370us remaining: 1.48ms
1: learn: 0.5616669 total: 670us remaining: 1.01ms
2: learn: 0.5343063 total: 980us remaining: 653us
3: learn: 0.5145676 total: 1.25ms remaining: 311us
4: learn: 0.4871167 total: 1.53ms remaining: 0us
Set the number of iterations to 5 (num_features=8)
Learning rate set to 0.496568
0: learn: 0.6246017 total: 399us remaining: 1.6ms
1: learn: 0.5735251 total: 713us remaining: 1.07ms
2: learn: 0.5290100 total: 980us remaining: 653us
3: learn: 0.5063135 total: 1.28ms remaining: 320us
4: learn: 0.4824728 total: 1.6ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6619798 total: 822us remaining: 3.29ms
1: learn: 0.6391259 total: 1.59ms remaining: 2.39ms
2: learn: 0.6234188 total: 2.18ms remaining: 1.45ms
3: learn: 0.6082446 total: 2.74ms remaining: 685us
4: learn: 0.5913410 total: 3.42ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6745133 total: 560us remaining: 2.24ms
1: learn: 0.6477514 total: 775us remaining: 1.16ms
2: learn: 0.6168266 total: 1.03ms remaining: 688us
3: learn: 0.5912126 total: 1.25ms remaining: 312us
4: learn: 0.5766633 total: 1.45ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6722615 total: 893us remaining: 3.57ms
1: learn: 0.6504818 total: 2.46ms remaining: 3.68ms
2: learn: 0.6258208 total: 4.09ms remaining: 2.73ms
3: learn: 0.6140295 total: 4.99ms remaining: 1.25ms
4: learn: 0.5897780 total: 5.63ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6694875 total: 524us remaining: 2.1ms
1: learn: 0.6429623 total: 997us remaining: 1.5ms
2: learn: 0.6304890 total: 1.53ms remaining: 1.02ms
3: learn: 0.6198958 total: 1.95ms remaining: 487us
4: learn: 0.6078768 total: 2.29ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6659685 total: 709us remaining: 2.84ms
1: learn: 0.6528495 total: 1.65ms remaining: 2.47ms
2: learn: 0.6351691 total: 2.23ms remaining: 1.49ms
3: learn: 0.6255924 total: 2.82ms remaining: 704us
4: learn: 0.6010411 total: 3.41ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6560034 total: 502us remaining: 2.01ms
1: learn: 0.6341959 total: 895us remaining: 1.34ms
2: learn: 0.6121527 total: 1.46ms remaining: 974us
3: learn: 0.5856772 total: 2.16ms remaining: 540us
4: learn: 0.5636024 total: 2.66ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6638959 total: 588us remaining: 2.35ms
1: learn: 0.6450034 total: 1.02ms remaining: 1.52ms
2: learn: 0.6295305 total: 1.32ms remaining: 880us
3: learn: 0.5986613 total: 1.63ms remaining: 406us
4: learn: 0.5802317 total: 1.93ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6734583 total: 395us remaining: 1.58ms
1: learn: 0.6570375 total: 615us remaining: 923us
2: learn: 0.6450230 total: 913us remaining: 609us
3: learn: 0.6306584 total: 1.16ms remaining: 290us
4: learn: 0.6188438 total: 1.52ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6699863 total: 390us remaining: 1.56ms
1: learn: 0.6508914 total: 630us remaining: 945us
2: learn: 0.6195118 total: 841us remaining: 560us
3: learn: 0.6148326 total: 1.26ms remaining: 314us
4: learn: 0.5982021 total: 1.45ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6570992 total: 797us remaining: 3.19ms
1: learn: 0.6170805 total: 1.4ms remaining: 2.09ms
2: learn: 0.5995415 total: 1.98ms remaining: 1.32ms
3: learn: 0.5899890 total: 2.5ms remaining: 625us
4: learn: 0.5678127 total: 3.38ms remaining: 0us
Set the number of iterations to 5 (num_features=6)
Learning rate set to 0.496568
0: learn: 0.6753442 total: 545us remaining: 2.18ms
1: learn: 0.6578389 total: 959us remaining: 1.44ms
2: learn: 0.6283958 total: 1.25ms remaining: 834us
3: learn: 0.6029148 total: 1.53ms remaining: 381us
4: learn: 0.5836538 total: 1.87ms remaining: 0us
[ ]: