RNN problems with multi-feature and separate label set (in Tensorflow) - recurrent-neural-network

Question

I'm learning to use a RNN to predict market index, e.g. S&P500 (note; that's the S&P index, not 500 different companies). Aside from price change data I'm also feeding in other info such as RSI, MACD, EMA
I then have 3 labels which are 1w, 2w 3w in future which I load from a separate CSV.
Say this is my example data (completely made-up):
price change RSI MACD EMA
0.3 3.2 0.1 0.0
-0.1 3.1 0.1 0.0
-1.2 3.8 0.1 0.2
0.9 2.7 0.1 0.2
1.3 1.7 0.2 0.2
I then have a separate CSV for the labels
1w future price change % 2w future price change % 3w future price change %
1.2 1.8 -0.3
0.8 0.2 1.1
0.2 1.5 0.7
1.2 1.7 0.1
-0.2 1.8 -0.3
My trouble is I can only find examples that use single features and/or use future training data as the labels, whereas I use a separate defined set of data for the labels.
I've cobble together the code below, but I get a shape feed error on this line:
mse = loss.eval(feed_dict={X: trX, Y: trY})
I suspect the format of my data is wrong as it's still in the format I used to train a 'normal' feed-forward network. I suspect some re-shaping is needed, but to honest, I've not a clue into what format due to having multiples features. I might also have defined the model incorrectly(?).
I'd be grateful if someone could help me with this.
I've also an additional question: Previously (as you'll see in the code) I would shuffle the data, which would be fine for mini batch feed-forward NN, but how would that work with an RNN, where I assume you need to present the data in sequential order? Following on from that, lets say I adapted this for stocks (rather than market index); would I need to present data on a stock-by-stock basis to make up the moving window, rather than doing it on a day-to-day basis? Obviously on day-to-day basis each row of data would be for a different stock.
Sorry for all the question, still getting my head around RNNs!
import tensorflow as tf
import numpy as np
import pandas as pd
import datetime
from sklearn.model_selection import train_test_split
# hyperparameters
epochs = 600
batch_size = 128
num_hidden = 100
df = pd.read_csv('C:\\python\\MarketData-Inputs.csv',header=None)
ldf = pd.read_csv('C:\\python\\MarketData-Results.csv',header=None)
# 20% test, shuffle the data, and use random state for like-like comparison between runs
trX, teX, trY, teY = train_test_split(df, ldf, test_size=0.2, shuffle=True, random_state=42)
trX = trX.values.astype('float')
trY = trY.values.astype('float')
teX = teX.values.astype('float')
teY = teY.values.astype('float')
print(trX.shape)
print(trY.shape)
print(teX.shape)
print(teY.shape)
#data params
features_size = len(trX[0])
labels_size = len(trY[0])
step_size = 3
tf.reset_default_graph()
X = tf.placeholder("float", [None, step_size, features_size], name="X")
Y = tf.placeholder("float", [None, labels_size], name="Y")
basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=num_hidden, activation=tf.nn.relu)
rnn_outputs, _ = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)
stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, num_hidden])
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, labels_size)
outputs = tf.reshape(stacked_outputs, [-1, step_size, labels_size])
with tf.name_scope("loss"):
loss = tf.reduce_sum(tf.square(outputs - Y))
training_op = tf.train.AdamOptimizer().minimize(loss)
tf.summary.scalar("loss", loss)
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
for ep in range(epochs):
sess.run(training_op, feed_dict={X: trX, Y: trY})
if ep % 100 == 0:
mse = loss.eval(feed_dict={X: trX, Y: trY})
print(ep, "\tMSE:", mse)
y_pred = sess.run(stacked_outputs, feed_dict={X: teX})
print(y_pred)

Answer 1

Okay, I think I sussed it. In my example I'm not batching the data, but RNN expects an input tensor of at least rank 3, so I need to add an extra dimension to my input data to give the form of [batch,sequence,feature size], which I can do here:
trX = np.expand_dims( trX.values.astype('float'), axis=0)
trY = trY.values.astype('float')
teX = np.expand_dims( teX.values.astype('float'), axis=0)
teY = teY.values.astype('float')
and my placeholders look like this.
X = tf.placeholder("float", [None, sequence_size, features_size], name="X")
Y = tf.placeholder("float", [None, labels_size], name="Y")
Regarding my question about shuffling data: I think I can now also answer my own question; the sequence of data should remain because the order of the sequence is important for RNN (that's the whole point of them). Instead, data can be shuffled via batching. So a batch is made up of random subsets of our entire dataset. So if entire dataset is 1000 time steps, and I have a batch size of 100, then I could create 10 batches, and then present the batches in random/shuffled order to the RNN.
Hope this helps someone in future.