i have a content like this ( can be stored in a text file or database or etc):
[
{
"site_slug": "/site1",
"site_url": "http://site1.com"
}
]
i want nginx can read the data and map the location path with key: "site_slug" and redirect to "site_url"
is it possible? could you give me some keywords for googling?
Related
I´d like to dynamically generate a sitemap.xml containing all static and dynamic user links (through uids from Firestore) with Cloud Functions when a user or a crawler requests https://www.example.com/sitemap.xml. I already managed to implement a working version using sitemap.js (https://github.com/ekalinin/sitemap.js#generate-a-one-time-sitemap-from-a-list-of-urls) and Firebase Hosting rewrites. However, my current solution (see below) generates one large sitemap.xml and only works for up to 50000 links which is not scalable.
Current solution:
Hosting rewrite in firebase.json:
"hosting": [
...
"rewrites": [
{
"source": "/sitemap.xml",
"function": "generate_sitemap"
},
]
}
],
Function in index.ts
export const generateSitemap = functions.region('us-central1').https.onRequest((req, res) => {
const afStore = admin.firestore();
const promiseArray: Promise<any>[] = [];
const stream = new SitemapStream({ hostname: 'https://www.example.com' });
const fixedLinks: any[] = [
{ url: `/start/`, changefreq: 'hourly', priority: 1 },
{ url: `/help/`, changefreq: 'weekly', priority: 1 }
];
const userLinks: any[] = [];
promiseArray.push(afStore.collection('users').where('active', '==', true).get().then(querySnapshot => {
querySnapshot.forEach(doc => {
if (doc.exists) {
userLinks.push({ url: `/user/${doc.id}`, changefreq: 'daily', priority: 1 });
}
});
}));
return Promise.all(promiseArray).then(() => {
const array = fixedLinks.concat(userLinks);
return streamToPromise(Readable.from(array).pipe(stream)).then((data: any) => {
res.set('Content-Type', 'text/xml');
res.status(200).send(data.toString());
return;
});
});
});
Since, this scales only to about 50000 links, I´d like to do something like https://github.com/ekalinin/sitemap.js#create-sitemap-and-index-files-from-one-large-list. But it seems like I´d need to actually create and temporarily store .xml files somehow.
Does anyone have experience with this issue?
As you noted, this isn't scalable and your costs are going to skyrocket since you pay per read/write on Firestore, so I would recommend rethinking your architecture.
I solved a similar problem several years ago for an App Engine website that needed to generate sitemaps for millions of dynamically created pages and it was so efficient that it never exceeded the free tier's limits.
Step 1: Google Storage instead of Firestore
When a page is created, append that URL to a text file in a Google Storage bucket on its own line. If your URLs have a unique ID you can use that to search and replace existing URLs.
https://www.example.com/foo/some-long-title
https://www.example.com/bar/some-longer-title
If may be helpful to break the URLs into smaller files. If some URLs start with /foo and others start with /bar I'd create at least two files called sitemap_foo.txt and sitemap_bar.txt and store the URLs into their respective files.
Step 2: Dynamically Generate Sitemap Index
Instead of a normal enormous XML sitemap, create a sitemap index that points to your multiple sitemap files.
When /sitemap.xml is visited have the following index generated by looping through the sitemap files in your bucket and listing them like this:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://storage.google...../sitemap_foo.txt</loc>
</sitemap>
<sitemap>
<loc>https://storage.google...../sitemap_bar.txt</loc>
</sitemap>
</sitemapindex>
Step 3: Remove Broken URLs
Update your 404 controller to search and remove the URL from your sitemap if found.
Summary
With the above system you'll have a scalable, reliable and efficient sitemap generation system that will probably cost you little to nothing to operate.
Answers to your questions
Q: How many URLs can you have in a sitemap?
A: According to Google, 50,000 or 50MB uncompressed.
Q: Do I need to update my sitemap everytime I add a new user/post/page?
A: Yes.
Q: How do you write to a single text file without collisions?
A: Collisions are possible, but how many new pages/posts/users are being created per second? If more than one per second I would create a Pub/Sub topic with a function that drains it to update the sitemaps in batches. Otherwise I'd just have it update directly.
Q: Let's say I created a sitemap_users.txt for all of my users...
A: Depending on how many users you have, it may be wise to break it up even further to group them into users per month/week/day. So you'd have sitemap_users_20200214.txt that would contain all users created that day. That would most likely prevent the 50,000 URL limit.
I need to execute a Firebase function AND load a page using a single URL.
e.g.
URL = "localhost:5000/cars?make=hyundai&model=elantra"
This URL will load the views/cars/index.html webpage.
This URL will also call the loadCar() Firebase function.
...How can I achieve these two goals? (or is there a better way to achieve this - load a page AND invoke a function)
Current behaviour: It loads the page (achieve 1), but doesn't invoke the loadCar() function (doesn't achieve 2).
FYI my firebase.json:
{
"hosting": {
"public": "functions/views",
"rewrites": [
{
"source": "/cars/**",
"function": "loadCar"
}
]
}
My file directory (arrows pointing to relevant files):
loadCar():
exports.loadCar= functions.https.onRequest((req, res) => {
// const requestedCar = Get requested car info from URL queries.
// Save requestedCar to Firestore DB, so it can be loaded in the html view.
// ... etc.
res.end();
});
If you're asking if it's possible for a single Firebase Hosting URL to both load static content and trigger a function, it's not possible. It has to be either one, not both.
You could instead have some JavaScript code in the static content invoke the function through another URL. Or you could have the URL invoke the function, which returns HTML to display, in addition to performing other work. But the destination of the requested URL can only go to static content or a function, not both.
Referencing to Kestrel documentation is it possible to configure https using appsettings.json file:
"HttpsInlineCertStore": {
"Url": "https://+:5002",
"Certificate": {
"Subject": "<coma separated multi-line subject name>",
"Store": "Root",
"Location": "LocalMachine"
}
This certificate exist for sure and next code returns finds it:
using (var certStore = new X509Store(StoreName.Root, StoreLocation.LocalMachine))
{
certStore.Open(OpenFlags.ReadOnly);
var certificates = certStore.Certificates.Find(
X509FindType.FindBySubjectDistinguishedName, "<coma separated multi-line subject name>", true);
return certificates .Count > 0 ? certificates [0] : null;;
}
At the same time if to search certificate by X509FindType.FindBySubjectName it founds nothing and I believe this is the issue even though microsoft says that FindBySubjectDistinguishedName is more specific search.
Finally I was able to fix this issue:
is something like "CN=name, C=UK, ..." but if you want to FindBySubjectName you must remove "CN=" from search string and leave only the name so it is looks not like "CN=name" but like "name".
I have used the following code to set my reset email subject:
Accounts.emailTemplates.resetPassword.subject = function(user, url) {
var ul = Meteor.absoluteUrl();
var myArray = ul.split("//");
var array = myArray[1].split('/');
return "How to reset your password on "+array[0];
};
I want it to contain the current browser's url, but it's not happening.
This is what the subject looks like
How to reset your password on 139.59.9.214
but the desired outcome is:
How to reset your password on someName.com
where someName.com is my URL.
I would recommend handling this a bit differently. Your host name is tied to your environment, and depending on what your production environment looks like, deriving your hostname from the server might not always be the easiest thing to do (especially if you're behind proxies, load balancers, etc.). You could instead look into leveraging Meteor's Meteor.settings functionality, and create a settings file for each environment with a matching hostname setting. For example:
1) Create a settings_local.json file with the following contents:
{
"private": {
"hostname": "localhost:3000"
}
}
2) Create a settings.json file with the following contents:
{
"private": {
"hostname": "somename.com"
}
}
3) Adjust your code to look like:
Accounts.emailTemplates.resetPassword.subject = function (user, url) {
const hostname = Meteor.settings.private.hostname;
return `How to reset your password on ${hostname}`;
};
4) When working locally, start meteor like:
meteor --settings=settings_local.json
5) When deploying to production, make sure the contents or your settings.json file are taken into consideration. How you do this depends on how you're deploying to your prod environment. If using mup for example, it will automatically look for a settings.json to use in production. MDG's Galaxy will do the same.
How can I list all files inside a folder with Meteor.I have FS collection and cfs:filesystem installed on my app. I didn't find it in the doc.
Another way of doing this is by adding the shelljs npm module.
To add npm modules see: https://github.com/meteorhacks/npm
Then you just need to do something like:
var shell = Meteor.npmRequire('shelljs');
var list = shell.ls('/yourfolder');
Shelljs docs:
https://github.com/arturadib/shelljs
The short answer is that FS.Collection creates a Mongo collection that you can treat like any other, i.e., you can list entries using find().
The long answer...
Using cfs:filesystem, you can create a mongo database that mirrors a given folder on the server, like so:
// in lib/files.js
files = new FS.Collection("my_files", {
stores: [new FS.Store.FileSystem("my_files", {"~/test"})] // creates a ~/test folder at the home directory of your server and will put files there on insert
});
You can then access this collection on the client to upload files to the server to the ~test/ directory:
files.insert(new File(['Test file contents'], 'my_test_file'));
And then you can list the files on the server like so:
files.find(); // returns [ { createdByTransform: true,
_id: 't6NoXZZdx6hmJDEQh',
original:
{ name: 'my_test_file',
updatedAt: (Date)
size: (N),
type: '' },
uploadedAt: (Date),
copies: { my_files: [Object] },
collectionName: 'my_files'
}
The copies object appears to contain the actual names of the files created, e.g.,
files.findOne().copies
{
"my_files" : {
"name" : "testy1",
"type" : "",
"size" : 6,
"key" : "my_files-t6NoXZZdx6hmJDEQh-my_test_file", // This is the name of the file on the server at ~/test/
"updatedAt" : ISODate("2015-03-29T16:53:33Z"),
"createdAt" : ISODate("2015-03-29T16:53:33Z")
}
}
The problem with this approach is that it only tracks the changes made through the Collection; if you add something manually to the ~/test directory, it won't get mirrored into the Collection. For instance, if on the server I run something like...
mkfile 1k ~/test/my_files-aaaaaaaaaa-manually-created
Then I look for it in the collection, it won't be there:
files.findOne({"original.name": {$regex: ".*manually.*"}}) // returns undefined
If you just want a straightforward list of files on the server, you might consider just running an ls. From https://gentlenode.com/journal/meteor-14-execute-a-unix-command/33 you can execute any arbitrary UNIX command using Node's child_process.exec(). You can access the app root directory with process.env.PWD (from this question). So in the end if you wanted to list all the files in your public directory, for instance, you might do something like this:
exec = Npm.require('child_process').exec;
console.log("This is the root dir:");
console.log(process.env.PWD); // running from localhost returns: /Users/me/meteor_apps/test
child = exec('ls -la ' + process.env.PWD + '/public', function(error, stdout, stderr) {
// Fill in this callback with whatever you actually want to do with the information
console.log('stdout: ' + stdout);
console.log('stderr: ' + stderr);
if(error !== null) {
console.log('exec error: ' + error);
}
});
This will have to run on the server, so if you want the information on the client, you'll have to put it in a method. This is also pretty insecure, depending on how you structure it, so you'd want to think about how to stop people from listing all the files and folders on your server, or worse -- running arbitrary execs.
Which method you choose probably depends on what you're really trying to accomplish.