I was able to create a website with a domain name behind a ipfs gateway like cloudflare.
Can I listen HTTP post requests ?
A ipfs website like torrent-paradise.ml, seems to send HTTP post requests (ex: /api/search?q=test)
IIUC the website you mentioned uses old school nodejs app for the /api/search endpoint.
Search feature is not provided by the IPFS daemon.
By default IPFS gateway allows only HTTP GET.
One can enable experimental Writable Gateway feature, which then accepts HTTP POST: https://discuss.ipfs.io/t/writeable-http-gateways/210?u=lidel, but it only allows you to import data to IPFS. There is no search feature.
That being said, I believe you should not care about HTTP method used, but ask "how to build dynamic app on IPFS" or "how to do search using immutable data on IPFS" instead.
Some pointers/ideas:
Build a DAG-based index and put it on IPFS, then have your app traverse the graph while looking for answer (fetching only subset of the index, only when needed)
Leverage libp2p's pubsub for real-time features (eg. by running js-ipfs on the page)
Look into CRDTs for decentralized conflict-free data types
While you can do it all by hand and tailor your solution to a specific problem, reusable primitives for the last two are provided by existing projects built on top of IPFS, like OrbitDB.
Related
Need a tool/setup in place that will trigger RESTful webAPI requests using a preformatted file (which contains Http request config and payload) .. like the .http file format supported by VScode REST client.
We need a CLI because the process is automated and runs in background. A typical scenario will be a .http file arriving in a folder. The tool will pick it up (usually there will be a single Http Request) and trigger that Http request. The response will be output to another file (not mandatory but a logging feature would be helpful in debugging).
Here're a few options we've been exploring -
Postman Newman
CURL with a windows Batch file (or use HttPie instead of curl)
Similar tools we've been exploring - VScode REST client, httpYAC, .. (not sure if such tools can be automated)
A console app based solution - using Node.js, C#, Python, php, ... it'll be highly customizable but this will be like start from scratch. So this is the last option if all the above fails.
We just need this piece to complete our data flow. And we're yet to face things like request throttle, delay, auth, ... but all of this needs to be preconfigured and automated. The setup can very based on the Http service provider we use (i.e. Shopify, Amazon, ...).
EDITT #1 :
Option#5. Forgot to mention that we had implemented a webAPI access demo using SQL SPs (OLE Automation) and we could achieve a lot of what we wanted. Here's a similar Ref.
We are trying to implement our product with SOA and currently using IBM Integration Bus v9 as our ESB.
We have 3 different environments (Sets of servers used for different purposes) that we deploy our product on:
development: Used during the testing and development process
customer test: More stable builds for customer's approval before going for the main release
main/production: This is the final thing.
The challenge we have encountered: Setting Base URL for HTTP nodes of our message flows for different environments; without compromising the DRY principle!
It seems that it's only possible to set the whole URL at once in HTTP Request nodes with the mqsiapplybaroverride command. The problem is that multiple resources can be exposed from a single server and thus have a common base URL.
Using UDP seems to be a promising approach. We can set base URL for each of our services in a UDP and build HTTP Request URLs in compute nodes just before the HTTP Request node using the UDP. Then the UDPs can be overrided with the mqsiapplybaroverride... Problem? It seems that UDPs don't have a scope of more than a single message flow... so anytime I want to call a resource from a server I have to define a UDP for that message flow or the BAR override won't affect the base URL for that message flow... This would lead to base URLs being repeated in each message flow... DRY applies.
This should be a common problem in a typical SOA application... So is there any better way to solve it? Anything like JNDI feature in typical Java EE Containers?
IIB v10.0.0.6 seems to have introduced a RestRequest node which provides Base URL setting capability... Unfortunately, we don't have that luxury for the time being.
You can use a user defined configurable service to achieve this.
You can read and set URL from configurable service using a java compute node or using a mapping node and custom java.
A good solution is to set baseUrl in database for each environment and set value as bellow:
SET OutputLocalEnvironment.Destination.REST.Request.BaseURL = GetCachedOrFromDB('custom_service_baseUrl');
GetCachedOrFromDB -> define function to get value from cache or db (if not in cache)
'custom_service_baseUrl' - > define property as key in settings table for each environment.
I have my oracle apex database application built in a oracle web hosting server. For ex, it is like http://www.myapexhostingserver.com/pls/apex/f?p=4550:1:13225322000472
Here I have created many apex database applications. Now I have specific domains for each of these apex applications. Currently, I have written a redirect in each of these domain settings to my apex application.
For ex,
http://www.domain1.com has a redirect url set to http://myapexhostingserver.com/pls/apex/f?p=162:LOGIN:16888048444232
http://www.domain2.com has a redirect url set to http://myapexhostingserver.com/pls/apex/f?p=165:LOGIN:16888048444232
A redirect is not google friendly. But I cannot purchase oracle web hosting for each domain too. Is there any other good solution available?
Meaning, when I buy a shared hosting server, I can create as many PHP applications I want. But not sure if this is also possible in oracle apex.
Can someone help?
Developing Oracle Apex Applications with Search Engine Visible Designs
This is a discussion on the issues with the varying style of APEX Application url syntax. There also is an explanation of how APEX work space namespaces are organized to show how application url's can be customized using a single database instance.
Comments: Some Internet searches can yield an answer that you're looking for. There are a number of developers and consulting gigs out there who have tackled the problem of converting between the popular, canonical url paths and the notation used by Oracle APEX.
Some search terms that proved useful include: oracle apex pretty url, url mapping, and http url mapping plsql. The most useful guides were examples in Oracle PL/SQL, but you can also dive deeper and set up rules on the web server engine (i.e., Apache HTTP Server) that re-interprets URL requests so that the APEX styled addresses are masked. This isn't necessary, but some techniques such as "url remapping" are among the possible solutions.
What this post is NOT is a guide on specifics of Search Engine Optimization (SEO), which may be distantly related to the OP's question about "Google Friendliness", but is beyond the scope of this solution guide.
Are Oracle APEX Websites Search Engine Friendly?
A comment quoted directly from the original post was worth researching and addressing. This is mostly an incorrect conclusion.
A redirect is not google friendly.
This actually depends on how you set things up on your APEX website. Redirects are one way of managing incoming traffic that use alternate url naming schemes, but this is not the only approach.
Some Successful Blogs and Websites Done in Apex:
Some online groups such as the APEX developer team with APEX Ninjas continue to report growing referrals from search engines such as Google without many changes to the way their Oracle APEX powered site and blog.
Another good example is the active APEX website: "ASK Tom", which was created by Oracle's Tom Kyte. This site has run on APEX for many years now and references to pages on this website continue to appear on search engine result pages the familiar APEX style url formatting from the "ASK Tom" website.
Using Better Navigation URLs for Clarity
Web server REDIRECTS are one approach to mapping cleaner url address syntax on top of an Oracle Apex Listener. We use the Abyss Web Server Engine, which is a light-weight http server that sits on the front of our company APEX application server. It's not absolutely necessary, but we chose it for its simplified management of a singlular web server task. Redirects work fine for us as well. Though we have not explored further, advanced features such as virtual hosting and url rewriting rules may take you further regarding your requirement to separate out multiple TLDs (primary domains) or C-NAMEs (sub-domains) on the same Apex listener instance.
The ugliest of the URL forms in APEX Applications is the default. It is decipherable, but not friendly to the human eye or simple enough to remember. Here is an example shot of the URL assigned to the HOME page of my application named: APEX-LAB-PUB
This is a common format of the URLs that an Oracle APEX web server will interpret. So the general format to observe is that everything after the clause: f?p=...
http://somedomain.com/pls/apex/f?p=1312786:5:987529480
By using the alternative option of setting ALIAS names, you can create short-cut url forms in the same layout, but with human readable alias assignments:
This is good, but there is also a common practice which uses Oracle Apex's native REST Web Service listener so that internal and external links look simpler. Simplified url syntax can also make setting up redirect hooks cleaner and more organized.
Oraganizing APEX Application Websites Using REST Web Services
To develop this example, I used Oracle's free Apex hosting tier. Although it has some limitations on what features are available, RESTful web services are enabled on the Apex listener of this platform, which is what we need.
Mapping a Web Service URI to an Apex URL Request
This is the url syntax we will be using will look something like this:
https://apex.oracle.com/pls/apex/...(workspace)/...(uri request)
Uniform Resource Identifier (URI)
This is the part that will be appended to the end of the base url of the Oracle Apex hosting listener. The example below is just a generic template:
Generic URI Syntax This Solution URI
hr/employees/{ID} display/{app}/{page}
|_|_________|____| |______|____________|
| | | | |
| | -Handler Bind Variable | -Handler Bind Variable(s)
| | |
| - URI Template - URI Template
|
- Module URI Prefix
-- RESTful Service Design Summary
Module Name: superEasyURL
URI Prefix (optional): (none)
URI Template: display/{app}/{page}
Handler Method: GET
Handler Source Type: PL/SQL
Handler Source:
declare
url varchar2(500);
baseurl constant varchar2(2000):=
'https://apex.oracle.com/pls/apex/f?p=';
begin
url:= baseurl || :app || ':' || :page;
:URL := url;
:STATUS := 303;
end;
Note: You will also need to set two OUT parameters for this REST Service to represent the two bind variables in the above PL/SQL statement for the values of URL and STATUS:
About the HTTP 303 Status Header Code
There is another kind of HTTP status code called an HTTP 303. You can use APEX to deliver pages and content that are not redirects. Oracle Application Express and its RESTful Web services can receive traditional url requests that convert to the APEX format on the back-end.
303 See Other has been proposed as one way of responding to a request for a URI that identifies a real-world object according to Semantic Web theory (the other being the use of hash URIs).
For example, if http://www.example.com/id/alice identifies a person, Alice, then it would be inappropriate for a server to respond to a GET request with 200 OK, as the server could not deliver Alice herself. Instead the server would issue a 303 See Other response which redirected to a separate URI providing a description of the person Alice.
Reference Excerpt: Wikipedia.com
Demonstration of a Working Example
This is an example created in my personal, Oracle hosted workspace. This is the url which will be interpreted by the REST Service built earlier.
https://apex.oracle.com/pls/apex/rgpascual/display/{app}/{page}
|_________|
|
|
- Workspace ID String (required by the listener) -
Note that on the shared instance, an additional reference to the user's workspace name was required to make this translation happen. I am not 100% sure if it is required, but I was able to complete this demo by using it in the final path references.
Display of: HOME page (aliased) from the APEX-LAB-PUB Application (aliased as: APEXLABPUB01).
Conclusions, Questions and General Advice
The efforts of this post offers the possibility of using pretty url syntax to represent navigation links within an Apex Application website. Instead of the default notation defined by Oracle, a navigation link list may look like:
Note: These links do not work. They are just to illustrate the advantages of this presented solution.
Home
Actual Requested URL:
someapexserver.../pls/apex/mywork/display/new-app/home
About Us
Actual Requested URL:
someapexserver.../pls/apex/mywork/display/new-app/about
Contact
Actual Requested URL:
someapexserver.../pls/apex/mywork/display/new-app/contactus
Associates
Actual Requested URL:
someapexserver.../pls/apex/mywork/display/new-app/associates
Business Ops
Actual Requested URL:
someapexserver.../pls/apex/mywork/display/new-app/businessops
Now, referring links and navigation is uniform and consistent using the URI template chosen in the initial design of this solution.
Additional Questions: Application Capacity of a Single Apex Instance
There was an additional question about the way that Oracle Apex manages applications created in a single instance.
... Meaning, when I buy a shared hosting server, I can create as many PHP applications I want. But not sure if this is also possible in oracle apex.
You should consider reviewing the Apex documentation put out by Oracle to understand how the Apex security framework affects the possibilities or organizing multiple applications created from the same database instance.
A few things to note however:
A single database installation, or INSTANCE can contain multiple divisions called WORKSPACES. Workspaces by default cannot see what is in other workspaces but applications within may share access to the same database schemas on the back-end.
A WORKSPACE may contain any number of APEX applications. Each application enabled with login security may have its own user authentication or it may use the "native" authentication process. Without further configuration, Application User accounts in the same Workspace will have have access to the other apps in the same workspace.
The USER NAMESPACES of each Workspace are separate, i.e. there may be two user accounts with the same name between two or more different Workspaces. (a user named ADMIN in both WORKSPACE1 and WORKSPACE2 are NOT the same user.)
NAMED DATABASE USERS may have access across other multiple Apex workspaces. A user defined this way is unique across every workspace and application permitting its access. Access users defined as Named Database Users are defined unique across a given database INSTANCE.
So in conclusion, Apex is designed to support multiple applications similarly to hosting on a single, shared hosting set up. Chances are however, if you are looking for the amount of control you describe in the OP, your shared hosting will most likely be your own Dedicated Server solution, VPS (Virtual Private Server) or AWS (Amazon Web Services) host.
You will need a host with enough access to server resources to: (a) Permit the installation of Oracle database software on the host. (b) Allow the configuration (opening and/or closing) of server networking ports. (c) Enough server resources to support the active operation of the installed database instance and the web-requests serviced by the accompanying HTTP based listener.
If you want your installation to manage multiple moderately sized and trafficked web applications, you will need a lot more than the minimum server specifications described in the installation instructions of the Oracle database version and edition you choose.
I want to let my clients speak directly with ElasticSearch REST API, obviously preventing them from performing any data or configuration change.
I had a look at ElasticSearch REST interface and I noticed the pattern: HTTP GET requests are pretty safe (harmless queries and status of cluster).
So I thought I can use Cloudfront as a CDN/Proxy that only allows GET/HEAD methods (you can impose such restrict it in the main configuration).
So far so good, all is set up. But things don't work because I would need to open my EC2 security group to the world in order to be reachable from Cloudfront! I don't want this, really!
When I use EC2 with RDS, I can simply allow access to my EC2 security group in RDS security groups. Why can't I do this with CloudFront? Or can I?
Ideas?
edit: It's not documented, but ES accepts facets query, which involve a (JSON) body, not only with POST, but also with GET. This simply breaks HTTP recommendation (as for RFC3616) by not ignoring the body for GET request (source).
This relates because, as pointed out, exposing ES REST interface directly can lead to easy DOS attacks using complex queries. I'm still convinced though, having one less proxy is still worth it.
edit: Other option for me would be to skip CloudFront and adding a security layer as an ElasticSearch plugin as shown here
I ended coding with my own plugin. Surprisingly there was nothing quite like this around.
No proxies, no Jetty, no Tomcat.
Just a the original ES rest module and my RestFilter. Using a minimum of reflection to obtain the remote address of the requests.
enjoy:
https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin
Note that even a GET request can be harmful in Elasticsearch. A query which simply takes up too much resources to compute will bring down your cluster. Facets are a good way to do this.
I'd recommend writing a simple REST API you place in front of ES so you get much more control over what hits your search cluster. If that's not an option you could consider running Nginx on your ES boxes to act as a local reverse proxy, which will give you the same control (and a whole lot more) as CloudFront does. Then you'd only have to open up Nginx to the world, instead of ES.
A way to do this in AWS would be:
Set up an Application Load Balancer in front of your ES cluster. Create a TLS cert for the ALB and serve https. Open the ES security group to the ALB.
Set up CloudFront and use the ALB as origin. Pass a custom header with a secret value (for WAF, see next point).
Set up WAF on your ALB to only allow requests that contain the custom header with the secret value. Now all requests have to go through CloudFront.
Set up a Lambda#Edge function on your CloudFront distribution to either remove the body from GET requests, or DENY such requests.
It’s quite some work, but there’s advantages over the plugin, e.g.:
CloudFront comes with free network DDOS protection
CloudFront gives your users lower latency to ES because of the fast CloudFront network and global PoP’s.
Opens many options to use CloudFront, WAF and Lamba#Edge to further protect your ES cluster.
I’m working on sample code in CDK to set all of this up. Will report back when that’s ready.
Someone has developed some OpenTSDB application to interface with HTTP API Rest?
I'm new both to OpenTSDB and both to HTTP programming.
Check out openTSDB2.0 documentation : opneTSDB documetation It has better support for adding and scanning nodes via HTTP api.
It sounds like you are looking for OpenTSDB clients. You will find a list of these on the Resources page.
If you want to monitor generic server statistics, these might already be implemented by one of the utilities, like tcollector. We use scollector, which is a part of Bosun (see the Resources page above).
Check the Documentation for OpenTSDB HTTP API. Scroll down to API Endpoints section, you will find enpoint URLs for put,aggregators,query,search, etc