Btukfyl

I have been stumped on this problem for a very long time and cannot figure it out. I believe the issue stems from subsets of data.frame objects retaining information of the parent but I also feel it's causing issues when training h2o.deeplearning models on what I think is just my training set (though this may not be true). See below for sample code. I included comments to clarify what I'm doing but it's fairly short code:

dataset = read.csv("dataset.csv")[,-1] # Read dataset in but omit the first column (it's just an index from the original data)

y = dataset[,1] # Create response

X = dataset[,-1] # Create regressors



X = model.matrix(y~.,data=dataset) # Automatically create dummy variables

y=as.factor(y) # Ensure y has factor data type

dataset = data.frame(y,X) # Create final data.frame dataset



train = sample(length(y),length(y)/1.66) # Create training indices -- A boolean

test = (-train) # Create testing indices



h2o.init(nthreads=2) # Initiate h2o



# BELOW: Create h2o.deeplearning model with subset of dataset.

mlModel = h2o.deeplearning(y='y',training_frame=as.h2o(dataset[train,,drop=TRUE]),activation="Rectifier",

                           hidden=c(6,6),epochs=10,train_samples_per_iteration = -2)





predictions = h2o.predict(mlModel,newdata=as.h2o(dataset[test,-1])) # Predict using mlModel

predictions = as.data.frame(predictions) # Convert predictions to dataframe object. as.vector() caused issues for me

predictions = predictions[,1] # Extract predictions



mean(predictions!=y[test]) 

The problem is that if I evaluate this against my test subset I get almost 0% error:

[1] 0.0007531255

Has anyone encountered this issue? Have an idea of how to alleviate this problem?

It will be more efficient to use the H2O functions to load the data and split it.

data = h2o.importFile("dataset.csv")

y = 2 #Response is 2nd column, first is an index

x = 3:(ncol(data))  #Learn from all the other columns

data[,y] = as.factor(data[,y])



parts = h2o.splitFrame(data, 0.8)  #Split 80/20

train = parts[[1]]

test = parts[[2]]



# BELOW: Create h2o.deeplearning model with subset of dataset.

mlModel = h2o.deeplearning(x=x, y=y, training_frame=train,activation="Rectifier",

                           hidden=c(6,6),epochs=10,train_samples_per_iteration = -2)



h2o.performance(mlModel, test)

It is hard to say what the problem with your original code is, without seeing the contents of dataset.csv and being able to try it. My guess is that train and test are not being split, and it is actually being trained on the test data.