Artifact of #6

Code

# Breiman (2001) Table 2, row 'Liver', column 'Selection' (paper: 25.1 %). UCI BUPA liver disorders, 345 rows, 6 inputs, class = selector field.
# Protocol of Section 4: random 10% hold-out; forests of 100 trees with F = 1 and F = int(log2 M + 1) (M = 6, so F = 3);
# the test error of the forest with the lower out-of-bag error is kept (ties go to F = 1); repeated 100 times and averaged. One average = one seed.
import sys, math, urllib.request
import numpy as np
from sklearn.ensemble import RandomForestClassifier
URL = "https://archive.ics.uci.edu/ml/machine-learning-databases/liver-disorders/bupa.data"
raw = urllib.request.urlopen(URL, timeout=60).read().decode()
rows = [r.split(",") for r in raw.strip().splitlines()]
X = np.array([[float(v) for v in r[:6]] for r in rows]); y = np.array([int(r[6]) for r in rows])
n, M = X.shape
F2 = int(math.log2(M) + 1)
def one_rep(rng):
    idx = rng.permutation(n); nt = n // 10
    te, tr = idx[:nt], idx[nt:]
    best = None
    for F in (1, F2):
        rf = RandomForestClassifier(n_estimators=100, max_features=F, bootstrap=True, oob_score=True,
                                    n_jobs=-1, random_state=int(rng.integers(2**31 - 1)))
        rf.fit(X[tr], y[tr])
        oob_err = 1 - rf.oob_score_
        te_err = np.mean(rf.predict(X[te]) != y[te])
        if best is None or oob_err < best[0]: best = (oob_err, te_err)
    return best[1]
seeds = int(sys.argv[1]) if len(sys.argv) > 1 else 4
print("rows", n, "inputs", M, "F2", F2, "test cases per repetition", n // 10)
res = []
for seed in range(seeds):
    rng = np.random.default_rng(seed)
    e = 100 * np.mean([one_rep(rng) for _ in range(100)])
    res.append(e); print(f"seed {seed}: mean test error over 100 repetitions = {e:.2f}%")
print(f"mean over {seeds} seeds {np.mean(res):.2f}%  sd across seeds {np.std(res, ddof=1):.2f}")

Expected output

rows 345 inputs 6 F2 3 test cases per repetition 34
seed 0: mean test error over 100 repetitions = 26.56%
seed 1: mean test error over 100 repetitions = 26.97%
seed 2: mean test error over 100 repetitions = 27.82%
seed 3: mean test error over 100 repetitions = 26.47%
seed 4: mean test error over 100 repetitions = 26.62%
seed 5: mean test error over 100 repetitions = 26.47%
seed 6: mean test error over 100 repetitions = 27.74%
seed 7: mean test error over 100 repetitions = 26.65%
mean over 8 seeds 26.91%  sd across seeds 0.56

Verifications (0)

Nobody has run it yet.