Flask/NGINX 404s on only some POST requests - nginx

I have a Flask app served by Gunicorn using NGINX as a webserver. I only have two endpoints that have POST requests, a login and a submit post. The login works flawlessly however whenever I attempt to POST using the submit post endpoint, I receive a 404. My code works running on localhost without NGINX or Gunicorn and in order to detect that the request is POST vs GET I am using WTForms validate_on_submit method. The NGINX error and access logs don't show anything out of the ordinary.
Edit: Here is the code for the endpoints with post requests
New Post:
#app.route('/newpost', methods=['GET', 'POST'])
#login_required
def new_post():
form = BlogForm()
if form.validate_on_submit():
#Omitted code for getting form data and creating form object
try:
#Omitted code for database actions
return redirect(url_for('index', _external=True))
except IntegrityError:
flash("Error inserting into database")
return('', 203)
else:
#Omitted some code for template rendering
return render_template('new_post.html')
Login:
#app.route('/login', methods=['GET', 'POST'])
def login():
form = LoginForm()
userAuth = current_user.is_authenticated
if form.validate_on_submit():
flash("Login attempt logged")
user = User.query.filter_by(username=form.userID.data).first()
if user:
checkPass = form.password.data.encode('utf-8')
if bcrypt.checkpw(checkPass, user.password):
user.authenticated = True
login_user(user, remember=form.remember_me.data)
return redirect(url_for('index', _external=True))
else:
form.errors['password'] = 'false'
else:
form.errors['username'] = 'false'
else:
return render_template('login.html',
title='Log In',
loggedIn=userAuth,
form=form)

The problem ended up being with NGINX after all, I fixed it by changing the ownership of /var/lib/nginx from the nginx user to the user running the Flask application. Prior to doing this AWS wasn't letting me upload files from my form in new_post.

Related

couldn't authenticate spotify in vercel production

I made a spotify clone which have a login,main pages. The user is initially directed to login page (localhost:3000/login) and once the user clicks login button they are taken to spotify authentication callback url (from spotify side, we can login using google, facebook or email etc) and once the user is logged in (successfully authenticated) , spotify provides a token which is used to check if the user is authenticated from client side. If success, the user is taken to the main page (which has all the music in it).
The spotify dashboard takes in redirect url for authentication which in my case its -> http://localhost:3000/api/auth/callback/spotify
This workflow worked perfectly when working and running locally ( localhost:3000 ).
When hosting in vercel,
I created a project in vercel without envs
I took my https://sp-app.vercel.app which is the domain and added to NEXTAUTH_URL into my env, and also added - NEXT_PUBLIC_CLIENT_SECRET, NEXT_PUBLIC_CLIENT_ID and JWT_SECRET
Went to spotify dashboard and editing the redirect url to https://sp-app.vercel.app/api/auth/callback/spotify and saved
redeployed the app from vercel (which gave me a deployent url, but nevermind) and clicked on the domains --> sp-app.vercel.app
The login page came, once I clicked on login button, it loads and stays in login page itself. It isn't moving to my home page nor authenticating but this same code worked locally fine.
Code to understand :
Login button:
{providers !== null && Object.values(providers).map((provider) => (
<div key={provider.name}>
<button onClick={()=>signIn(provider.id, {callbackUrl:"/"})}
>Login with {provider.name}</button>
</div>
}
export async function getServerSideProps(){
const providers = await getProviders(); //getProviders is imported from next-auth
return {
props:{
providers,
}
}
}
middleware:
export async function middleware(req){
const token = await getToken({req,secret:process.env.JWT_SECRET});
const {pathname} = req.nextUrl;
//Allow the request if,
// (1) Its a request for next-auth session & provider fetching
// (2) The token exists
if(pathname.includes('/api/auth') || token){
return NextResponse.next();
}
//Redirect to login if no token or requesting a protected route
if(!token && pathname !== "/login"){
return NextResponse.redirect("/login");
}
}
While I checked with Network tab, I get a session which should log me in and redirect to main page, but it isn't working in vercel deployment.
Hey so first off make sure your NEXTAUTH_URL environment variable is correct. Also check that the redirect URI (from the Spotify developer dashboard) is correct.
After that, add this secureCookie code to the getToken function in the _middleware.js file. So that function should end up looking like this:
const token = await getToken({
req,
secret: process.env.JWT_SECRET,
secureCookie:
process.env.NEXTAUTH_URL?.startsWith("https://") ??
!!process.env.VERCEL_URL,
});
Let me know if this works. I had the same exact issue and this worked for me.

python fastapi redirect and authenticate

I'm trying to authenticate the user after visiting the registration link
(link example: http://127.0.0.1:8000/confirm-email?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9F)
My code:
#app.get("/confirm-email", status_code=200, )
def confirm_email(
token: str = fastapi.Query(..., min_length=64, max_length=256,
db: Session = fastapi.Depends(database.get_db)):
if user := crud.read_user_by(db, column='current_token', value=token):
if user.created_datetime + timedelta(minutes=30) > datetime.now(): # TODO change minutes to days
return fastapi.responses.RedirectResponse(
url="http://127.0.0.1:8000/users/me",
headers={"access_token": token, "token_type": "bearer"})
else:
raise fastapi.HTTPException(
status_code=fastapi.status.HTTP_410_GONE,
detail="Confirmation link is expired")
else:
raise fastapi.HTTPException(
status_code=fastapi.status.HTTP_401_UNAUTHORIZED,
detail="Wrong token")
#app.get("/users/me")
def read_users_me(token: str = fastapi.Depends(oauth2_scheme),
db: Session = fastapi.Depends(database.get_db)):
try:
return services.get_user_by_token(db=db, token=token)
except Exception as e:
raise fastapi.HTTPException(
status_code=fastapi.status.HTTP_401_UNAUTHORIZED,
detail="Could not validate credentials",
headers={"WWW-Authenticate": "Bearer"},
)
But every time I'm failing when trying to use /users/me endpoint (getting 401 error, UNAUTHORIZED).
Maybe I put the token in the wrong place or using wrong headers?
If using OAuth 2.0 and wanting to set the access_token in a request, tipically, it goes into the Authorization header like the example in the RFC: Authorization: Bearer mF_9.B5f-4.1JqM - in the example, mF_9.B5f-4.1JqM would be the value of the token.
It seems to me that you are accessing the users/me endpoint with the headers access_token: [token value] and token_type: "bearer". Instead, I believe the following header should be set: Authorization: Bearer [token value]
After a little researching, I figured out that redirection by specification can't have authorization headers (browser/client will just ignore it mainly). So even if headers are correct - it's nonsense. One possible solution to use URL.

Login working in Splash API but not when SplashRequest is used

Relatively new to Splash. I'm trying to scrape a website which needs a login. I started off with the Splash API for which I was able to login perfectly. However, when I put my code in a scrapy spider script, using SplashRequest, it's not able to login.
import scrapy
from scrapy_splash import SplashRequest
class Payer1Spider(scrapy.Spider):
name = "payer1"
start_url = "https://provider.wellcare.com/provider/claims/search"
lua_script = """
function main(splash,args)
assert(splash:go(args.url))
splash:wait(0.5)
local search_input = splash:select('#Username')
search_input:send_text('')
local search_input = splash:select('#Password')
search_input:send_text('')
assert(splash:wait(0.5))
local login_button = splash:select('#btnSubmit')
login_button:mouse_click()
assert(splash:wait(7))
return{splash:html()}
end
"""
def start_requests(self):
yield SplashRequest(self.start_url, self.parse_result,args={'lua_source': self.lua_script},)
def parse_result(self, response):
yield {'doc_title' : response.text}
The output HTML is the login page and not the one after logging in.
You have to add endpoint='execute' to your SplashRequest to execute the lua-script:
yield SplashRequest(self.start_url, self.parse_result, args={'lua_source': self.lua_script}, endpoint='execute')
I believe you don't need splash to login to the site indeed. You can try next:
Get https://provider.wellcare.com and then..
# Get request verification token..
token = response.css('input[name=__RequestVerificationToken]::attr(value)').get()
# Forge post request payload...
data = [
('__RequestVerificationToken', token),
('Username', 'user'),
('Password', 'pass'),
('ReturnUrl', '/provider/claims/search'),
]
#Make dict from list of tuples
formdata=dict(data)
# And then execute request
scrapy.FormRequest(
url='https://provider.wellcare.com/api/sitecore/Login',
formdata=formdata
)
Not completely sure if all of this will work. But you can try.

Symfony2 + FosOauth: token get, but requests are anonymous

I have installed and configurated FosOauthBundle but I have this problem: I can get Token And Refresh token with this line of code:
$ http POST http://localhost:8888/app_dev.php/oauth/v2/token grant_type=password client_id=1_3bcbxd9e24g0gk4swg0kwgcwg4o8k8g4g888kwc44gcc0gwwk4 client_secret=4ok2x70rlfokc8g0wws8c8kwcokw80k44sg48goc0ok4w0so0k username=MYEMAIL password=MYPASS
If I pass a wrong MYEMAIL or a wrong MYPASS symfony reply me with an error (and this is correct). Users are manged via FosUserBundle.
Now, How can I use token to say symfony that I'm a registered user ?
I have created this "api" in my controller:
public function getDemosAction()
{
$user = $this->get('security.token_storage')->getToken()->getUser();
$view = $this->view($user);
return $this->handleView($view);
}
and I call it via this code:
$curl -H "Authorization: Bearer NGEwMGIxMzJkZmU5Yjc3YmM2ZjViNmE0YWFhYTEwOTg1MjI5NzIyNDkwNmFhYTUzMTRkZTk3MzEyNjA4OWY0Ng" http://localhost:8888/app_dev.php/api/demos
but it return me ALWAYS "anon.".
Where is my error ?
The problem was the order of firewall directives in security.yml. I have found the solution via this other question: Symfony2 OAuth keeps giving me a login page when a token is provided

HTTP test server accepting GET/POST requests

I need a live test server that accepts my requests for basic information via HTTP GET and also allows me to POST (even if it's really not doing anything). This is entirely for test purposes.
A good example is here. It easily accepts GET requests, but I need one that accepts POST requests as well.
Does anyone know of a server that I can send dummy test messages too?
https://httpbin.org/
It echoes the data used in your request for any of these types:
https://httpbin.org/anything Returns most of the below.
https://httpbin.org/ip Returns Origin IP.
https://httpbin.org/user-agent Returns user-agent.
https://httpbin.org/headers Returns header dict.
https://httpbin.org/get Returns GET data.
https://httpbin.org/post Returns POST data.
https://httpbin.org/put Returns PUT data.
https://httpbin.org/delete Returns DELETE data
https://httpbin.org/gzip Returns gzip-encoded data.
https://httpbin.org/status/:code Returns given HTTP Status code.
https://httpbin.org/response-headers?key=val Returns given response headers.
https://httpbin.org/redirect/:n 302 Redirects n times.
https://httpbin.org/relative-redirect/:n 302 Relative redirects n times.
https://httpbin.org/cookies Returns cookie data.
https://httpbin.org/cookies/set/:name/:value Sets a simple cookie.
https://httpbin.org/basic-auth/:user/:passwd Challenges HTTPBasic Auth.
https://httpbin.org/hidden-basic-auth/:user/:passwd 404'd BasicAuth.
https://httpbin.org/digest-auth/:qop/:user/:passwd Challenges HTTP Digest Auth.
https://httpbin.org/stream/:n Streams n–100 lines.
https://httpbin.org/delay/:n Delays responding for n–10 seconds.
There is http://ptsv2.com/
"Here you will find a server which receives any POST you wish to give it and stores the contents for you to review."
Webhook Tester is a great tool: https://webhook.site (GitHub)
Important for me, it showed the IP of the requester, which is helpful when you need to whitelist an IP address but aren't sure what it is.
nc one-liner local test server
Setup a local test server in one line under Linux:
nc -kdl localhost 8000
Sample request maker on another shell:
wget http://localhost:8000
then on the first shell you see the request showed up:
GET / HTTP/1.1
User-Agent: Wget/1.19.4 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: localhost:8000
Connection: Keep-Alive
nc from the netcat-openbsd package is widely available and pre-installed on Ubuntu.
Tested on Ubuntu 18.04.
http://requestb.in was similar to the already mentioned tools and also had a very nice UI.
RequestBin gives you a URL that will collect requests made to it and let you inspect them in a human-friendly way.
Use RequestBin to see what your HTTP client is sending or to inspect and debug webhook requests.
Though it has been discontinued as of Mar 21, 2018.
We have discontinued the publicly hosted version of RequestBin due to ongoing abuse that made it very difficult to keep the site up reliably. Please see instructions for setting up your own self-hosted instance.
If you want a local test server that accepts any URL and just dumps the request to the console, you can use node:
const http = require("http");
const hostname = "0.0.0.0";
const port = 3000;
const server = http.createServer((req, res) => {
console.log(`\n${req.method} ${req.url}`);
console.log(req.headers);
req.on("data", function(chunk) {
console.log("BODY: " + chunk);
});
res.statusCode = 200;
res.setHeader("Content-Type", "text/plain");
res.end("Hello World\n");
});
server.listen(port, hostname, () => {
console.log(`Server running at http://localhost:${port}/`);
});
Save it in a file 'echo.js' and run it as follows:
$ node echo.js
Server running at http://localhost:3000/
You can then submit data:
$ curl -d "[1,2,3]" -XPOST http://localhost:3000/foo/bar
which will be shown in the server's stdout:
POST /foo/bar
{ host: 'localhost:3000',
'user-agent': 'curl/7.54.1',
accept: '*/*',
'content-length': '7',
'content-type': 'application/x-www-form-urlencoded' }
BODY: [1,2,3]
Have a look at PutsReq, it's similar to the others, but it also allows you to write the responses you want using JavaScript.
Here is one Postman echo: https://docs.postman-echo.com/
example:
curl --request POST \
--url https://postman-echo.com/post \
--data 'This is expected to be sent back as part of response body.'
response:
{"args":{},"data":"","files":{},"form":{"This is expected to be sent back as part of response body.":""},"headers":{"host":"postman-echo.com","content-length":"58","accept":"*/*","content-type":"application/x-www-form-urlencoded","user-agent":"curl/7.54.0","x-forwarded-port":"443","x-forwarded-proto":"https"},"json":{"...
You can run the actual Ken Reitz's httpbin server locally (under docker or on bare metal):
https://github.com/postmanlabs/httpbin
Run dockerized
docker pull kennethreitz/httpbin
docker run -p 80:80 kennethreitz/httpbin
Run directly on your machine
## install dependencies
pip3 install gunicorn decorator httpbin werkzeug Flask flasgger brotlipy gevent meinheld six pyyaml
## start the server
gunicorn -b 0.0.0.0:8000 httpbin:app -k gevent
Now you have your personal httpbin instance running on http://0.0.0.0:8000 (visible to all of your LAN)
Minimal Flask REST server
I wanted a server which returns predefined responses so I found that in this case it's simpler to use a minimal Flask app:
#!/usr/bin/env python3
# Install dependencies:
# pip3 install flask
import json
from flask import Flask, request, jsonify
app = Flask(__name__)
#app.route('/', methods=['GET', 'POST'])
def root():
# spit back whatever was posted + the full env
return jsonify(
{
'request.json': request.json,
'request.values': request.values,
'env': json.loads(json.dumps(request.__dict__, sort_keys=True, default=str))
}
)
#app.route('/post', methods=['GET', 'POST'])
def post():
if not request.json:
return 'No JSON payload! Expecting POST!'
# return the literal POST-ed payload
return jsonify(
{
'payload': request.json,
}
)
#app.route('/users/<gid>', methods=['GET', 'POST'])
def users(gid):
# return a JSON list of users in a group
return jsonify([{'user_id': i,'group_id': gid } for i in range(42)])
#app.route('/healthcheck', methods=['GET'])
def healthcheck():
# return some JSON
return jsonify({'key': 'healthcheck', 'status': 200})
if __name__ == "__main__":
with app.test_request_context():
app.debug = True
app.run(debug=True, host='0.0.0.0', port=8000)
I don't konw why all of the answers here make a very simple work very hard!
When there is a request on HTTP, actually a client will send a HTTP_MESSAGE to server (read about what is HTTP_MESSAGE) and you can make a server in just 2 simple steps:
Install netcat:
In many unix-based systems you have this already installed and if you have windows just google it , the installation process is really simple, you just need a nc.exe file and then you should copy the path of this nc.exe file to your path environment variable and check if every thing is OK with nc -h
Make a server which is listening on localhost:12345:
just type nc -l -p 12345 on your terminal and everything is done! (in mac nc -l 12345 tnx Silvio Biasiol)
Now you have a server which is listening on http://localhost:12345 , for example you can make a post request with axios If you are a js developer:
axios.post('http://localhost:12345', { firstName: 'Fred' })
or make your own xhr or make a form in a HTML file and submit it to server, sth. like:
<form action="http://localhost:12345" method="post">
or make a request with curl or wget or etc. Then check your terminal, a raw HTTP_MESSAGE should be appear on your terminal and you can start your happy hacking ;)
https://www.mockable.io. It has nice feature of getting endpoints without login (24h temporary account)
Create choose a free web host and put the following code
<h1>Request Headers</h1>
<?php
$headers = apache_request_headers();
foreach ($headers as $header => $value) {
echo "<b>$header:</b> $value <br />\n";
}
?>
I have created an open-source hackable local testing server that you can get running in minutes. You can create new API's, define your own response and hack it in any ways you wish to.
Github Link : https://github.com/prabodhprakash/localTestingServer
some online httpbin:
https://httpbin.org/
https://httpbingo.org/
https://quic.aiortc.org/httpbin/
get client ip, port, ua..
http://ifconfig.io/
get client ip, isp
https://www.cip.cc/
If you need or want a simple HTTP server with the following:
Can be run locally or in a network sealed from the public Internet
Has some basic auth
Handles POST requests
I built one on top of the excellent SimpleHTTPAuthServer already on PyPI. This adds handling of POST requests:
https://github.com/arielampol/SimpleHTTPAuthServerWithPOST
Otherwise, all the other options publicly available are already so good and robust.
You might don't need any web site for that, only open up the browser, press F12 to get access to developer tools > console, then in console write some JavaScript Code to do that.
Here I share some ways to accomplish that:
For GET request:
*.Using jQuery:
$.get("http://someurl/status/?messageid=597574445", function(data, status){
console.log(data, status);
});
For POST request:
Using jQuery $.ajax:
var url= "http://someurl/",
api_key = "6136-bc16-49fb-bacb-802358",
token1 = "Just for test",
result;
$.ajax({
url: url,
type: "POST",
data: {
api_key: api_key,
token1: token1
},
}).done(function(result) {
console.log("done successfuly", result);
}).fail(function(error) {
console.log(error.responseText, error);
});
Using jQuery, append and submit
var merchantId = "AA86E",
token = "4107120133142729",
url = "https://payment.com/Index";
var form = `<form id="send-by-post" method="post" action="${url}">
<input id="token" type="hidden" name="token" value="${merchantId}"/>
<input id="merchantId" name="merchantId" type="hidden" value="${token}"/>
<button type="submit" >Pay</button>
</div>
</form> `;
$('body').append(form);
$("#send-by-post").submit();//Or $(form).appendTo("body").submit();
Using Pure JavaScript:
`var api_key = "73736-bc16-49fb-bacb-643e58",
recipient = "095552565",
token1 = "4458",
url = 'http://smspanel.com/send/';`
``var form = `<form id="send-by-post" method="post" action="${url}">
<input id="api_key" type="hidden" name="api_key" value="${api_key}"/>
<input id="recipient" type="hidden" name="recipient" value="${recipient}"/>
<input id="token1" name="token1" type="hidden" value="${token1}"/>
<button type="submit" >Send</button>
</div>
</form>`;``
document.querySelector("body").insertAdjacentHTML('beforeend',form);
document.querySelector("#send-by-post").submit();
Or even using ASP.Net:
var url = "https://Payment.com/index";
Response.Clear();
var sb = new System.Text.StringBuilder();
sb.Append("<html>");
sb.AppendFormat("<body onload='document.forms[0].submit()'>");
sb.AppendFormat("<form action='{0}' method='post'>", url);
sb.AppendFormat("<input type='hidden' name='merchantId' value='{0}'>", "C668");
sb.AppendFormat("<input type='hidden' name='Token' value='{0}'>", "22720281459");
sb.Append("</form>");
sb.Append("</body>");
sb.Append("</html>");
Response.Write(sb.ToString());
Response.End();
I am not sure if anyone would take this much pain to test GET and POST calls. I took Python Flask module and wrote a function that does something similar to what #Robert shared.
from flask import Flask, request
app = Flask(__name__)
#app.route('/method', methods=['GET', 'POST'])
#app.route('/method/<wish>', methods=['GET', 'POST'])
def method_used(wish=None):
if request.method == 'GET':
if wish:
if wish in dir(request):
ans = None
s = "ans = str(request.%s)" % wish
exec s
return ans
else:
return 'This wish is not available. The following are the available wishes: %s' % [method for method in dir(request) if '_' not in method]
else:
return 'This is just a GET method'
else:
return "You are using POST"
When I run this, this follows:
C:\Python27\python.exe E:/Arindam/Projects/Flask_Practice/first.py
* Restarting with stat
* Debugger is active!
* Debugger PIN: 581-155-269
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
Now lets try some calls. I am using the browser.
http://127.0.0.1:5000/method
This is just a GET method
http://127.0.0.1:5000/method/NotCorrect
This wish is not available. The following are the available wishes:
['application', 'args', 'authorization', 'blueprint', 'charset', 'close', 'cookies', 'data', 'date', 'endpoint', 'environ', 'files', 'form', 'headers', 'host', 'json', 'method', 'mimetype', 'module', 'path', 'pragma', 'range', 'referrer', 'scheme', 'shallow', 'stream', 'url', 'values']
http://127.0.0.1:5000/method/environ
{'wsgi.multiprocess': False, 'HTTP_COOKIE': 'csrftoken=YFKYYZl3DtqEJJBwUlap28bLG1T4Cyuq', 'SERVER_SOFTWARE': 'Werkzeug/0.12.2', 'SCRIPT_NAME': '', 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/method/environ', 'SERVER_PROTOCOL': 'HTTP/1.1', 'QUERY_STRING': '', 'werkzeug.server.shutdown': , 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36', 'HTTP_CONNECTION': 'keep-alive', 'SERVER_NAME': '127.0.0.1', 'REMOTE_PORT': 49569, 'wsgi.url_scheme': 'http', 'SERVER_PORT': '5000', 'werkzeug.request': , 'wsgi.input': , 'HTTP_HOST': '127.0.0.1:5000', 'wsgi.multithread': False, 'HTTP_UPGRADE_INSECURE_REQUESTS': '1', 'HTTP_ACCEPT': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 'wsgi.version': (1, 0), 'wsgi.run_once': False, 'wsgi.errors': ", mode 'w' at 0x0000000002042150>", 'REMOTE_ADDR': '127.0.0.1', 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, sdch, br'}
Another one that offers some customization and is easy to use (no install, signup) is https://beeceptor.com .
You create a endpoint, makes a initial request to it and can tweak the responses.
I am using this REST API all the time: https://restful-api.dev
It stores the created objects indefinitely.
Also, the schema is quite flexible, you can pass any JSON data.
I am a Front-End developer and is very useful when I need to create some sample data. This is the only one I could find that does it for free without any registration or tokens.
Just set one up yourself. Copy this snippet to your webserver.
echo "<pre>";
print_r($_POST);
echo "</pre>";
Just post what you want to that page. Done.

Resources