Artifact of #8

Code

# Breiman (2001) Table 2, row 'Sonar', column 'Selection' (paper: 15.9 %). UCI sonar (connectionist bench), 208 rows, 60 inputs, class = last field (R or M).
# Protocol of Section 4: random 10% hold-out; forests of 100 trees with F = 1 and F = int(log2 M + 1) (M = 60, so F = 6);
# the test error of the forest with the lower out-of-bag error is kept (ties go to F = 1); repeated 100 times and averaged. One average = one seed.
import sys, math, urllib.request
import numpy as np
from sklearn.ensemble import RandomForestClassifier
URL = "https://archive.ics.uci.edu/ml/machine-learning-databases/undocumented/connectionist-bench/sonar/sonar.all-data"
raw = urllib.request.urlopen(URL, timeout=60).read().decode()
rows = [r.split(",") for r in raw.strip().splitlines()]
X = np.array([[float(v) for v in r[:60]] for r in rows]); y = np.array([1 if r[60] == "M" else 0 for r in rows])
n, M = X.shape
F2 = int(math.log2(M) + 1)
def one_rep(rng):
    idx = rng.permutation(n); nt = n // 10
    te, tr = idx[:nt], idx[nt:]
    best = None
    for F in (1, F2):
        rf = RandomForestClassifier(n_estimators=100, max_features=F, bootstrap=True, oob_score=True,
                                    n_jobs=-1, random_state=int(rng.integers(2**31 - 1)))
        rf.fit(X[tr], y[tr])
        oob_err = 1 - rf.oob_score_
        te_err = np.mean(rf.predict(X[te]) != y[te])
        if best is None or oob_err < best[0]: best = (oob_err, te_err)
    return best[1]
seeds = int(sys.argv[1]) if len(sys.argv) > 1 else 4
print("rows", n, "inputs", M, "F2", F2, "test cases per repetition", n // 10)
res = []
for seed in range(seeds):
    rng = np.random.default_rng(seed)
    e = 100 * np.mean([one_rep(rng) for _ in range(100)])
    res.append(e); print(f"seed {seed}: mean test error over 100 repetitions = {e:.2f}%")
print(f"mean over {seeds} seeds {np.mean(res):.2f}%  sd across seeds {np.std(res, ddof=1):.2f}")

Expected output

rows 208 inputs 60 F2 6 test cases per repetition 20
seed 0: mean test error over 100 repetitions = 17.70%
seed 1: mean test error over 100 repetitions = 15.90%
seed 2: mean test error over 100 repetitions = 17.45%
seed 3: mean test error over 100 repetitions = 16.80%
seed 4: mean test error over 100 repetitions = 16.00%
seed 5: mean test error over 100 repetitions = 15.90%
seed 6: mean test error over 100 repetitions = 16.45%
seed 7: mean test error over 100 repetitions = 19.25%
mean over 8 seeds 16.93%  sd across seeds 1.16

Verifications (0)

Nobody has run it yet.