Modifying Wordpress post status on Publish - wordpress

Question

I am attempting to validate fields on a custom post type in the admin panel Edit Post page.
When the user clicks "Publish" I want to validate fields in the POST data, and change the post_status to "pending" if the data does not pass the tests. When this occurs, I'd also like to add errors to the page in the admin notices area.
I've been trying this with an added hook to the "wp_insert_post" action which also saves our own data. I'm not certain of the order of operations, but I'm assuming that the wp_insert_post events happen first, and then my function gets called via the hook.
The problem is that it's the Wordpress function which is doing the post publish actions, so by the time I get to validate data, Wordpress has already saved the post with a status of "publish". What I need to do is either prevent that update, or change the status back to "pending", but I'm having little success in finding a way to do this within the API.
So, here's an order of operations I'd like to effect:
1. admin user edits post data and clicks "Publish"
2. via wp_insert_post, my data validation and post meta save routine is called
3. If data passes validation, post status is "published"
4. Otherwise, post status set to "pending" & message shown in admin notice area
Surely someone has done this, but extensive Googling just leads me to the same seemingly irrelevant pages. Can someone point me in the right direction here? Thanks in advance-
UPDATE:
So, RichardML was indeed correct, hooking to the wp_insert_post_data filter gave me the right place to validate admin post edit page fields. I'm updating this however to note what the rest of the solution is, specifically getting the reason reported in the admin notice area.
First off, you can't just output data or set a field because the admin page is the result of a redirect, and by the time you get to rendering the admin post page again, the admin_notices action is already gone. The trick was something I picked up from another forum, and it's hackish, but it works.
What you'll need to do is in your validation filter function, if you determine that you will need to display errors, is use set_option() to add a blog option with a unique name (I used 'publish_errors'). This should be HTML code in a div with a class of "error".
You will also need to add an action hook for 'admin_notices', pointing at a function which checks for the existence of the 'publish_errors' option, and if it finds it, prints it to the page and deletes it with delete_option().

Answer 1

You can use the wp_insert_post_data filter to inspect and modify post data before it's inserted into the database.
In response to your update I don't think it's necessary to temporarily add an option to the database. It should be possible to simply add a query string variable to the Wordpress redirect, something like this:
add_filter('wp_insert_post_data', 'my_post_data_validator', '99');
function my_post_data_validator($data) {
if ($data['post_type'] == 'post') {
// If post data is invalid then
$data['post_status'] = 'pending';
add_filter('redirect_post_location', 'my_post_redirect_filter', '99');
}
return $data;
}
function my_post_redirect_filter($location) {
remove_filter('redirect_post_location', __FILTER__, '99');
return add_query_arg('my_message', 1, $location);
}
add_action('admin_notices', 'my_post_admin_notices');
function my_post_admin_notices() {
if (!isset($_GET['my_message'])) return;
switch (absint($_GET['my_message'])) {
case 1:
$message = 'Invalid post data';
break;
default:
$message = 'Unexpected error';
}
echo '<div id="notice" class="error"><p>' . $message . '</p></div>';
}