OEmbed not applied to video link in Ajax Callback - wordpress

Question

I'm having difficulties with wp_editor(), tinyMCE and the_content filter in relation to oEmbed of video.
I am outputting a front end wp_editor() form to allow registered users to create new posts from the front end of the site. The new post created is a custom post type.
The target behaviour is:
The registered user enters content & clicks submit
The form is processed by jQuery/Ajax, with form data passed to a PHP function via post()
A new post created, and a response is generated for an Ajax callback
The response is a JSON array that contains the HTML of the new post content
The returned HTML has 'the_content' filter applied - embedded video should display properly
The Ajax callback removes the original form and appends the post HTML to a div
Everything works as expected with the exception of video oEmbed.
If a video link is added to the content (on a new line within the wp_editor), the content built by the Ajax callback includes the video URL wrapped in paragraph tags - oEmbed hasn't worked, even though the HTML has had 'the_content' filter applied.
Refreshing the page displays the new post in a loop, with content displayed by the_content() tag - and the video is displayed properly (oEmbed has worked).
Setting 'wpautop' => false in the wp_editor arguments doesn't help - messes up formatting, doesn't fix video.
Is there a tinyMCE setting that I'm missing?
Is there a problem with how I'm applying 'the_content' filter and/or building a HTML string for the Ajax callback?
Relevant code shown below.
Thanks!
JQuery
(function( $ ) { 'use strict';
$(function() {
$('#student-submission-button').click( function(event) {
// Prevent default action
// -----------------------
event.preventDefault();
var submission_nonce_id = $('#the_nonce_field').val();
var submission_title = $('#inputTitle').val();
tinyMCE.triggerSave();
var submission_content = $('#editor').val();
var Data = {
action: 'student_submission',
nonce: submission_nonce_id,
workbook_ID: submission_workbook_ID,
content: submission_content,
title: submission_title,
};
// Do AJAX request
$.post( ajax_url, Data, function(Response) {
if( Response ) {
var submissionStatus = Response.status;
var submissionMessage = Response.report;
var postHTML = Response.content;
if ( 'success' == submissionStatus ) {
$('#user-feedback').html( submissionMessage );
$('#new-post').append( postHTML );
}
// Hide the form
$('.carawebs-frontend-form').hide(800, function() {
$(this).remove();
});
}
});
});
});
})( jQuery );
PHP
/**
* Return data via Ajax (excerpt)
*
*
*/
$response = array();
if( is_int( $new_submission_ID ) ) {
// Build a success response
// ------------------------
$new_post = get_post( $new_submission_ID, OBJECT );
$new_post_content = $new_post->post_content;
$return_content = "<h2>$new_post->post_title</h2>";
$return_content .= apply_filters( 'the_content', $new_post_content );
$response['status'] = "success";
$response['report'] = "New post created, ID: $new_submission_ID";
$response['content'] = $return_content;
} else {
// error report
}
wp_send_json( $response ); // send $response as a JSON object
Form HTML and wp_editor()
<form action="<?php echo $_SERVER['REQUEST_URI']; ?>" method="post" enctype="multipart/form-data" class="carawebs-frontend-form">
<label for="inputTitle">Title</label>
<input type="text" class="form-control" id="inputTitle" name="inputTitle" placeholder="Title" value="" />
<label for="inputContent" class="topspace">Your Content</label>
<?php
$args = array(
'textarea_rows' => 45,
'teeny' => false,
'editor_height' => 400,
'editor_class' => 'cwfrontendadmin',
'quicktags' => false,
'textarea_name' => 'cw_content',
'tinymce' => array(
'content_css' => get_template_directory_uri() . '/assets/css/editor-style.css'
),
);
wp_editor( 'Enter your content...', 'editor', $args );
wp_nonce_field('name_of_action','the_nonce_field', true, true ); // name of action, name of nonce field
?>
<input id="student-submission-button" class="btn btn-primary" type="submit" name="submission-form" value="Save Content" />
Update
I've narrowed this down to the way that the_content filter is applied. I think filtered content is cached, so oEmbed may not get applied to all content if the post content is returned outside the loop.
Now I have video oEmbed working - using a different method of inserting post_content into a variable:
<?php
global $post;
$post = get_post($new_submission_ID);
setup_postdata( $post );
$new_content = apply_filters('the_content', get_the_content());
$new_post_link = get_the_permalink();
$new_post_title = get_the_title();
wp_reset_postdata( $post );
This works fine, but it would be good if someone could explain why the original method of building the HTML didn't work.

Answer 1

The oEmbed filter does not get applied if the post content is returned outside the loop, and global $post is not set.
This is because the video embed content is cached in the postmeta table, and is is inaccessible if the post content is returned outside the loop. The WP_Embed class which is hooked onto by the_content filter is not designed for use outside the loop - this is referenced on Trac here.
The following method returns a working Video oEmbed as per the original scenario. global $post needs to be set to make the cached video embed data available:
<?php
global $post;
$post = get_post( $new_submission_ID );
setup_postdata( $post );
$new_content = apply_filters( 'the_content', get_the_content() );
wp_reset_postdata( $post );
// return what you need via Ajax callback -
// $new_content contains the proper video embed HTML
TL;DR Version
If you need oEmbed filters to be applied outside the loop, you need to set the global post variable so that the WP_Embed class can access the cached video embed html in the post's meta.

Answer 2

The reason, according to other sources is because the WP_Embed::shortcode() (tasked with swapping out the video for the embedded html) is calling on the global $post variable for information.
This means you need to be sure that the global $post variable contains the information of the post you're requesting.
My AJAX response method now incorporates the global $post variable:
global $post;
$post = get_post( $id );
echo apply_filters( 'the_content', $post->post_content );
die();
This is essentially just an extension of your own findings, but I thought it might help to have a clear answer/solution to the problem of oEmbed + AJAX for WordPress.