I'm trying to use node-fetch to capture the contents of a page, and running into an unexpected error. I checked a similar question but it doesn't seem relevant. I am trying to fetch a HTTPS site using an HTTPS agent and agents, but I'm getting an unexpected error about HTTP. I wonder whether this may be due to redirects, but I can't see anything that would cause it. This only fails for this particular URL (works fine, for example, with https://www.robinhood.com) , and I'm trying to figure out why. Here is a minimal example. I'd note that this uses some certificates I have saved locally, but I'm not sure how necessary that is to reproduce.
//start SO example
var siteURL = "https://robinhood.com/l/privacy";
import path from 'path';
import sslrootcas from 'ssl-root-cas';
const rootCas = sslrootcas.create();
import {fileURLToPath} from 'url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
rootCas.addFile(path.resolve(__dirname,'intermediate.pem'));
import http from 'node:http';
import https from 'node:https';
import UserAgent from 'user-agents';
const myhttpsAgent = new https.Agent({ca: rootCas});
// const requestcheck = fetch("https://www.google.com", {
const requestcheck = fetch(siteURL, {
method: "GET"
,headers: {"User-Agent": new UserAgent() }
,agent: myhttpsAgent
})
Here is the error I'm getting:
node:internal/errors:477
ErrorCaptureStackTrace(err);
^
TypeError: Protocol "http:" not supported. Expected "https:"
at new NodeError (node:internal/errors:387:5)
at new ClientRequest (node:_http_client:177:11)
at request (node:http:96:10)
at file:///home/app/node_modules/node-fetch/src/index.js:94:20
at new Promise (<anonymous>)
at fetch (file:///home/app/node_modules/node-fetch/src/index.js:49:9)
at ClientRequest.<anonymous> (file:///home/app/node_modules/node-fetch/src/index.js:236:15)
at ClientRequest.emit (node:events:525:35)
at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:674:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
at TLSSocket.socketOnData (node:_http_client:521:22)
at TLSSocket.emit (node:events:525:35)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) {
code: 'ERR_INVALID_PROTOCOL'
}
I wonder whether this may be due to redirects, but I can't see anything that would cause it.
https://robinhood.com/l/privacy redirects to
https://robinhood.com/us/en/support/articles/privacy-policy which then redirects to
http://robinhood.com/us/en/support/articles/privacy-policy/
The latter URL is plain HTTP and thus the wrong protocol by a https-only user agent.
Related
Step 1: Automatically create a new Next.js project using the new beta app directory:
npx create-next-app#latest --experimental-app
pages/api/hello.ts
// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next'
type Data = {
name: string
}
export default function handler(
req: NextApiRequest,
res: NextApiResponse<Data>
) {
res.status(200).json({ name: 'John Doe' })
}
This file is identical to the one created automatically created by npx - there are no modifications.
I am trying to build a simple home page, which fetches data from the api which gets data from my database. Either way an await/async will be required. I am following the instructions from here.
In this example I will demonstrate that even awaiting the supplied hello api can't seem to run in production, and I can't work out why.
app/page.tsx
async function getHelloAsync() {
const res = await fetch('http://localhost:3000/api/hello', { cache: 'no-store' });
// The return value is *not* serialized
// You can return Date, Map, Set, etc.
// Recommendation: handle errors
if (!res.ok) {
// This will activate the closest `error.js` Error Boundary
throw new Error('Failed to fetch data');
}
return res.json();
}
export default async function Page() {
const hello = await getHelloAsync();
return (
<main>
<h1>Hello: {hello.name}</h1>
</main>
)
}
To test the hello api works, I confirm that running pn run dev and then curl http://localhost:3000/api/hello the following successful response is received:
{"name":"John Doe"}
Next up we exit the dev server and run:
pn run build
The first headache is that the build will completely fail to build unless one adds { cache: 'no-store' } to the fetch command:
const res = await fetch('http://localhost:3000/api/hello', { cache: 'no-store' });
or adds this to the top of app/page.tsx:
export const fetchCache = 'force-no-store';
I am actually not sure how one would even build this if you wanted to cache the response or use revalidate instead and provide an initial optimistic response, because without cache: no-store it refuses to build outright. Ideally instead it should just cache the result from /api/hello and not fail. Running the dev server at the same idea as doing the build does allow the build to work, but then as soon as you exit the dev server and run pn run start then all the api calls fail anyway. So that is not a good idea.
This leads us to the next problem - why are the api calls not working in production (i.e. when calling pn run start).
Step 2:
pn run build
pn run start
Confirm that the following still works and yes it does:
curl http://localhost:3000/api/hello
Result:
{"name":"John Doe"}
Now we visit http://localhost:3000 in a browser but, surprise! We get the following error:
> next start
ready - started server on 0.0.0.0:3000, url: http://localhost:3000
warn - You have enabled experimental feature (appDir) in next.config.js.
warn - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use at your own risk.
info - Thank you for testing `appDir` please leave your feedback at https://nextjs.link/app-feedback
(node:787) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11118:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async getHelloAsync (/Users/username/nextjstest/.next/server/app/page.js:229:17)
at async Page (/Users/username/nextjstest/.next/server/app/page.js:242:19) {
cause: Error: connect ECONNREFUSED ::1:3000
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16)
at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
errno: -61,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '::1',
port: 3000
}
}
[Error: An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included on this error instance which may provide additional details about the nature of the error.] {
digest: '3567993178'
}
Why is it saying that the connection is refused when we know the API is available? I can't get this to run at all. I know this is beta but surely the code should actually run right? How do I make this code work?
Also if anyone knows where where the logs are that I'm supposed to be accessing to see digest '3567993178' please let me know.
Background
We are developing a Slack Bot. This time we are using Bolt for JavaScript (Node.js) provided by Slack, React, Next.js, and ngrok. Here is what each of them does.
Bolt for JavaScript: I don't want to use Slack's bare-bones API, but want to benefit from the function that wraps it.
React: Needed to use Next.js
Next.js: Slack needs a request URL to notify my bot app when events such as mentions occur in Slack, but Next.js makes it easy to create an API endpoint to be set to that URL (e.g. /api/something)
ngrok: In the local development environment, that URL will be http://localhost:3000, so the protocol will be http instead of https. Slack does not allow this, so we need a URL that starts with https that tunnels to the local http://localhost:3000. ngrok provides that easily!
Problem to be solved.
I have already confirmed that if I type #xxxx in a certain workspace in Slack, the event is notified to https://xxxx.jp.ngrok.io/api/slack/events. However, in this API file
app.event("app_mention", async ({ event, say }) => {
.
.
.
}
is not invoked and the following error occurs
error - unhandledRejection: Error: listen EADDRINUSE: address already in use :::3000
I would like to know why and how to resolve this.
Source code
/api/slack/events.ts
import type { NextApiRequest, NextApiResponse } from "next";
require("dotenv").config();
import app from "../../../config/slackAuth";
export default async function handler(
req: NextApiRequest,
res: NextApiResponse
) {
// Unique case for Slack challenge
if (req.body.challenge) return res.status(200).json(req.body);
// Subscribe to 'app_mention' event in your App config
// See https://api.slack.com/tutorials/tracks/responding-to-app-mentions
app.event("app_mention", async ({ event, say }) => {
try {
// Response to the message in the thread where the event was triggered with #${message.user}
// See https://slack.dev/bolt-js/concepts#message-sending
await say({
text: `Hi <#${event.user}>!`,
thread_ts: event.ts,
});
} catch (error) {
await say({
text: `<#${event.user}> ${error.message}.`, // #userName Request failed with status code 429.
thread_ts: event.ts,
});
}
});
(async () => {
// Start this app
await app.start(process.env.PORT || 3000);
console.log("⚡️ Bolt app is running!");
})();
return res.status(404).json({ message: "Unknown event type" });
}
Error code
error - unhandledRejection: Error: listen EADDRINUSE: address already in use :::3000
at Server.setupListenHandle [as _listen2] (net.js:1331:16)
at listenInCluster (net.js:1379:12)
at Server.listen (net.js:1465:7)
at C:\Users\81906\Documents\slackGpt3\node_modules\#slack\bolt\dist\receivers\HTTPReceiver.js:176:25
at new Promise (<anonymous>)
at HTTPReceiver.start (C:\Users\81906\Documents\slackGpt3\node_modules\#slack\bolt\dist\receivers\HTTPReceiver.js:142:16)
at App.start (C:\Users\81906\Documents\slackGpt3\node_modules\#slack\bolt\dist\App.js:241:30)
at eval (webpack-internal:///(api)/./pages/api/slack/events.ts:69:69)
at handler (webpack-internal:///(api)/./pages/api/slack/events.ts:71:7)
at Object.apiResolver (C:\Users\81906\Documents\slackGpt3\node_modules\next\dist\server\api-utils\node.js:363:15) {
code: 'EADDRINUSE',
errno: -4091,
syscall: 'listen',
address: '::',
port: 3000
}
Issue
Using Slack Bolt for JavaScript with Next.js is not straightforward due to the following reasons:
Running npm run dev in a Next.js project starts a server at localhost:3000.
Running app.start() in Slack Bolt for JavaScript starts a server using Express.js, which also tries to use localhost:3000.
This causes an error because two servers are trying to use the same port.
This information was provided by someone at Slack, and the source can be found at https://github.com/slackapi/bolt-js/issues/1687.
Solution
You can change the port number used by Bolt to, for example, 3001.
However, this will make it difficult for the two servers at localhost:3000 and 3001 to communicate with each other.
The request URL registered in the Slack Bolt for JavaScript console is set to 3000, so events received there will not be able to flow to port 3001.
I'm using opencensus-python to track requests to my python fastapi application running in production, and exporting the information to Azure AppInsights using the opencensus exporters. I followed the Azure Monitor docs and was helped out by this issue post which puts all the necessary bits in a useful middleware class.
Only to realize later on that requests that caused the app to crash, i.e. unhandled 5xx type errors, would never be tracked, since the call to execute the logic for the request fails before any tracing happens. The Azure Monitor docs only talk about tracking exceptions through the logs, but this is separate from the tracing of requests, unless I'm missing something. I certainly wouldn't want to lose out on failed requests, these are super important to track! I'm accustomed to using the "Failures" tab in app insights to monitor any failing requests.
I figured the way to track these requests is to explicitly handle any internal exceptions using try/catch and export the trace, manually setting the result code to 500. But I found it really odd that there seems to be no documentation of this, on opencensus or Azure.
The problem I have now is: this middleware function is expected to pass back a "response" object, which fastapi then uses as a callable object down the line (not sure why) - but in the case where I caught an exception in the underlying processing (i.e. at await call_next(request)) I don't have any response to return. I tried returning None but this just causes further exceptions down the line (None is not callable).
Here is my version of the middleware class - its very similar to the issue post I linked, but I'm try/catching over await call_next(request) rather than just letting it fail unhanded. Scroll down to the final 5 lines of code to see that.
import logging
from fastapi import Request
from opencensus.trace import (
attributes_helper,
execution_context,
samplers,
)
from opencensus.ext.azure.trace_exporter import AzureExporter
from opencensus.trace import span as span_module
from opencensus.trace import tracer as tracer_module
from opencensus.trace import utils
from opencensus.trace.propagation import trace_context_http_header_format
from opencensus.ext.azure.log_exporter import AzureLogHandler
from starlette.types import ASGIApp
from src.settings import settings
HTTP_HOST = attributes_helper.COMMON_ATTRIBUTES["HTTP_HOST"]
HTTP_METHOD = attributes_helper.COMMON_ATTRIBUTES["HTTP_METHOD"]
HTTP_PATH = attributes_helper.COMMON_ATTRIBUTES["HTTP_PATH"]
HTTP_ROUTE = attributes_helper.COMMON_ATTRIBUTES["HTTP_ROUTE"]
HTTP_URL = attributes_helper.COMMON_ATTRIBUTES["HTTP_URL"]
HTTP_STATUS_CODE = attributes_helper.COMMON_ATTRIBUTES["HTTP_STATUS_CODE"]
module_logger = logging.getLogger(__name__)
module_logger.addHandler(AzureLogHandler(
connection_string=settings.appinsights_connection_string
))
class AppInsightsMiddleware:
"""
Middleware class to handle tracing of fastapi requests and exporting the data to AppInsights.
Most of the code here is copied from a github issue: https://github.com/census-instrumentation/opencensus-python/issues/1020
"""
def __init__(
self,
app: ASGIApp,
excludelist_paths=None,
excludelist_hostnames=None,
sampler=None,
exporter=None,
propagator=None,
) -> None:
self.app = app
self.excludelist_paths = excludelist_paths
self.excludelist_hostnames = excludelist_hostnames
self.sampler = sampler or samplers.AlwaysOnSampler()
self.propagator = (
propagator or trace_context_http_header_format.TraceContextPropagator()
)
self.exporter = exporter or AzureExporter(
connection_string=settings.appinsights_connection_string
)
async def __call__(self, request: Request, call_next):
# Do not trace if the url is in the exclude list
if utils.disable_tracing_url(str(request.url), self.excludelist_paths):
return await call_next(request)
try:
span_context = self.propagator.from_headers(request.headers)
tracer = tracer_module.Tracer(
span_context=span_context,
sampler=self.sampler,
exporter=self.exporter,
propagator=self.propagator,
)
except Exception:
module_logger.error("Failed to trace request", exc_info=True)
return await call_next(request)
try:
span = tracer.start_span()
span.span_kind = span_module.SpanKind.SERVER
span.name = "[{}]{}".format(request.method, request.url)
tracer.add_attribute_to_current_span(HTTP_HOST, request.url.hostname)
tracer.add_attribute_to_current_span(HTTP_METHOD, request.method)
tracer.add_attribute_to_current_span(HTTP_PATH, request.url.path)
tracer.add_attribute_to_current_span(HTTP_URL, str(request.url))
execution_context.set_opencensus_attr(
"excludelist_hostnames", self.excludelist_hostnames
)
except Exception: # pragma: NO COVER
module_logger.error("Failed to trace request", exc_info=True)
try:
response = await call_next(request)
tracer.add_attribute_to_current_span(HTTP_STATUS_CODE, response.status_code)
tracer.end_span()
return response
# Explicitly handle any internal exception here, and set status code to 500
except Exception as exception:
module_logger.exception(exception)
tracer.add_attribute_to_current_span(HTTP_STATUS_CODE, 500)
tracer.end_span()
return None
I then register this middleware class in main.py like so:
app.middleware("http")(AppInsightsMiddleware(app, sampler=samplers.AlwaysOnSampler()))
Explicitly handle any exception that may occur in processing the API request. That allows you to finish tracing the request, setting the status code to 500. You can then re-throw the exception to ensure that the application raises the expected exception.
try:
response = await call_next(request)
tracer.add_attribute_to_current_span(HTTP_STATUS_CODE, response.status_code)
tracer.end_span()
return response
# Explicitly handle any internal exception here, and set status code to 500
except Exception as exception:
module_logger.exception(exception)
tracer.add_attribute_to_current_span(HTTP_STATUS_CODE, 500)
tracer.end_span()
raise exception
I work with vue and go for frontend and backend respectively. I send post request to my server and get 403 error code message(notAllowed). But in postman I get the objects and is fine.
Vue and Vuex
My axios post request:
const response = await this.$axios.post(`http://localhost:8000/v1/org/${params.organization}/kkms/${params.kkm}/closeShift`,{
headers : {
'token' : this.state.token.value
}});
I know I should also use other properties like 'Content-Type' and etc in headers, but know it works well with only "token" property in the other requests. I want to know whether problem in backend or frontend?
It seems you have a mistake in the axios request.
You are receiving a 403, that means you are not authorized (or sometimes something else, check the comments in the question and down here ).
As can be found in axios docs, the post request looks like this:
axios.post(url[, data[, config]]).
It accepts the config (so the headers) as THIRD parameter, while you are setting it as second parameter. Add an empty FormData object as second param, and just shift your config to the third param.
const fakeData = new FormData();
const response = await this.$axios.post(`http://localhost:8000/v1/org/${params.organization}/kkms/${params.kkm}/closeShift`,
fakeData,
{
headers : {
'token' : this.state.token.value
}
});
I have deployed the ExpressJs app on firebase function and trying to trigger the express app with an HTTP trigger.
I am trying to authorize my express app with OAUTH2.0 to use QuickBooks API. However, I am getting the following error:
TypeError: Redirected path should match configured path, but got: /callback
at CodeFlow.getToken (/srv/node_modules/client-oauth2/src/client-oauth2.js:597:7)
at /srv/routes/callback.js:12:25
at Layer.handle [as handle_request] (/srv/node_modules/express/lib/router/layer.js:95:5)
at next (/srv/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/srv/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/srv/node_modules/express/lib/router/layer.js:95:5)
at /srv/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/srv/node_modules/express/lib/router/index.js:335:12)
at next (/srv/node_modules/express/lib/router/index.js:275:10)
at Function.handle (/srv/node_modules/express/lib/router/index.js:174:3)
The error is in following file:
callback.js
router.get('/', function (req, res) {
// Verify anti-forgery
console.log("OriginalURL "+req.originalUrl)
console.log("base URL "+req.baseUrl)
// Exchange auth code for access token
tools.intuitAuth.code.getToken(req.originalUrl).then(function (token) { //This is where promise is failing
I am using npm package called "client-oauth2".
client-oauth2.js
if (
typeof options.redirectUri === 'string' &&
typeof url.pathname === 'string' &&
url.pathname !== Url.parse(options.redirectUri).pathname
) {
return Promise.reject(
new TypeError('Redirected path should match configured path, but got: ' + url.pathname) //The source of Error. I am not able to find out the value of options.redirectUri
)
}
Following are my configuration details:
"redirectUri": "us-central1-thematic-runner-245505.cloudfunctions.net/createEstimate/callback"
I have made sure that this URL matches with Quickbooks side redirect URL. In the case of localhost it works just fine.
Localhost URL: http://localhost:5001/thematic-runner-245505/us-central1/createEstimate/
The value of req.originalUrl from callback.js is
OriginalURL /thematic-runner-245505/us-central1/createEstimate/callback?code=AB11585447829JwgpXMEpWOR6irwprMe9Z8aqRoSK4npFDKmte&state=Z0t9yRkl-dWaO2J5sJRDaTB9eZKvyyyVcHYQ&realmId=4620816365002391850
Could somebody help me with this error? I don't know what I am doing wrong in the case of production. The callback URL seems to be fine. Any help would be appreciated.
This isn't a valid URL:
"redirectUri": "us-central1-thematic-runner-245505.cloudfunctions.net/createEstimate/callback"
There's no scheme/protocol portion to that URL (e.g. https://). It should start with https:// but instead you're jumping right into the domain.
It needs to be a valid URL, including the protocol/scheme.