Goal
I want to make a request to a website with Python requests to scrape some information for containers location and time.
This is the website I'm trying to get data from : https://www.cma-cgm.com/ebusiness/tracking by inserting the container number.
I'm trying something simple, like :
import requests
url = "some_url_i_cant_find"
tracking_number = ABCD1234567
requests.post(url, payload=tracking_number)
Problem
I cannot find in the Network tab how the request to get the container's data is being processed.
I assume this has something to do with reCAPTCHA, but I don't know much about this or how to handle it.
Solution
Some other answer or topic regarding this issue
How to make a request to this website and read the response.
Related
I'm trying to learn properly how the http protocols are working.
I'm struggling to find online books or resources, as most of the time I'm finding how to make these requests in various languages and not how they actually work.
For context, I'm trying to build a flutter app with a rust server as an exercise, with some security standards. I've been programming for a while now, so I have some concepts settled down but I want to learn more about other stuff.
What I've understood for now is that it is possible to ask a server for something at some url, and from that url the server will send back html page content.
My questions are :
What data do the http requests carry ? can it be anything ? Or is it exclusive html text ?
When doing a http get request, is there anyway to write data in the body of the request ? or do I only have the url with params in it to ask the server for specific data ?
Can I write anything in a http request ? for example, encoded strings with private / public keys ?
How can the client be assured he is indeed talking to the right server when posting the first requests ?
Maybe I'm a little out of context for this forums, but I've been trying to learn all this properly and feel stuck with out I found online.
I'm using the requests library in python and here is my code, my goal is trying to get a soundcloud client ID
r = requests.get("https://soundcloud.com/search?q=a")
This URL is that I'm trying to get:
https://api-v2.soundcloud.com/me?client_id=[REDACTED]
Using print(r.url) I tried to get that URL but it shows the same one which is in the request. How could I make this not happen and make it actually give the last response url similar to loading a webpage and getting the last request with just the requests library? I've also tried r.json() which is just printing out the webpage HTML and is still on the same "url". Please help - I'm stuck at this.
Disclaimer: I am not good at understanding http requests, so please bear with me
I am trying to change the content of an html web page whenever an http GET/POST request is made. It would work something like this:
What I want to accomplish
When my phone is charging, it is going to send an http request to the web server. The web server is going to change the content of the webpage to say something like "Phone is charging."
What I've done so far
I managed to send an HTTP request from my phone to the server every time the phone connects to a charger, I just don't know what to do with the http request that arrives to the server.
Thanks ahead of time!
EDIT: I figured out, according to #LawrenceCherone (thanks Lawrence!) that I can't do this with a static html page and just nginx. He said that I have to use a scripting language. Does bash work for this? Or should I learn something like python, PHP or something else?
I can't find any tutorials online for what I am trying to accomplish for some reason. Haven't seen any tutorials on how to 'react' to a POST request
So I have a webpage, ("http://data.terapeak.com/verify/") and I don't see any & tags in the URL so I am unaware how to post data to this. I need to do this via HTTPRequest rather than browser control. I am creating a double threaded batch searching program. I have already successfully made this using a single browser control but that wont allow for multi-threading, atleast with my current knowledge due to the fact that even when creating a new frmBrw that already exists it needs for me to set the threat apartment to single. If i set it to single, I am unable to have it send the data the the excel sheet I need both threads to access. I hope this is clear... The basic question is how can I log into this form via HTTP request.
This isn't going to be easy to answer without further details however I suspect you'll need to provide the variables via a HTTP POST request.
Can you successfully login to this page in your browser? If so, run a proxy tool such as fiddler and check the HTTP headers it makes to the server. You should see the form variables being passed over. You then need to mimic this in code.
How to: Send Data Using the WebRequest Class
Hope this gets you started
Is there a way to go through a series of request and see what pages they are coming from? I am getting all HTTP requests sent from my PC. I am trying to see if there is a way in which I can just find out the main request. Like if a page has images on it, when the images request is sent, is there a way to see if the images are coming from another page using just HTTP requests. I don't know if I explained this well enough, so please ask any questions. I don't know if there is a way to do this, but I hope there is. Thanks!
If you're using Windows: Fiddler.