Why does Keras perform poorly on this simple toy dataset? - r

Question

Here I've created a toy dataset by randomly sampling from two bernoulli distributions dictated by the logistic functions
1 / (1 + exp(-0.2 * (x - 20)))
-1 / (1 + exp(-0.2 * (x - 80)))
My hope was that I could train a keras NNet with a 2-node hidden layer and a softmax activation function that would learn these two logistic functions, but the resulting model predicts probability of 1 for every x value.
library(keras)
train <- data.frame(
x = c(4.44, 8.25, 15.72, 17.53, 17.53, 17.86, 18.57, 20.22, 20.24, 20.57, 21.99, 25.06, 28.3, 31.1, 35.91, 37.29, 38.36, 39.58,
39.78, 40.1, 47.29, 51.67, 51.74, 53.52, 57.45, 62.69, 63.03, 69.03, 70.11, 74.44, 76.4, 79.81, 86.92, 87.59, 89.88),
y = c(0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0)
)
head(train, 10)
x y
1 4.44 0
2 8.25 0
3 15.72 0
4 17.53 0
5 17.53 0
6 17.86 0
7 18.57 0
8 20.22 0
9 20.24 1
10 20.57 1
# Build and fit model
model <- keras_model_sequential()
model <- layer_dense(object = model, input_shape = 1L, use_bias = TRUE, units = 2L, activation = 'sigmoid')
model <- layer_dense(object = model, units = 1L, activation = 'softmax', input_shape = 2L)
model <- compile(object = model, loss = 'binary_crossentropy', optimizer = 'sgd', metrics = c('accuracy'))
fit(object = model, x = dt$Age, y = dt$LittleSleep * 1, epochs = 30)
# Evaluate
predict_proba(object = model, x = train$x)[, 1]
[1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Why does Keras do such a poor job of fitting to the training data?

Answer 1

Keras is not doing a poor job, it is exactly doing the job you told it to do in your network architecture :)
You are using a softmax activation at the output with only one output neuron, meaning that the softmax will always output 1.0, as the output is normalized across neurons. Do not do that, use at least two output neurons so normalization can happen correctly.
As you use binary cross-entropy loss, a better choice of activation would be sigmoid at the output, which will work with a single output neuron.