I have an xls list of URLs that I would want to bulk download into a folder, but some of the URLs are so outdated that I need to retrieve its snapshots from webarchives.
Is there any way to do it in bulk using programming languages?
For now I know that by bulk adding "https://web.archive.org/web/20000000000000/" in front of any URL will give me the earliest archived snap shot from webarchives. I just want to know how to add this in front of all URLs in the spreadsheet, and then use some bulk downloader to do the job.
Related
I have recently migrated my website to WordPress and took this as a chance to improve the slugs and subfolder structure of roughly 900 posts and 200 pages. I've got an Excel document which lists these 1100 old (source) and new (target) URLs. There's no common pattern among all of them.
source URL
target URL
/random-example
/examples/random
/some-blog-post
/news/something
/other-example
/examples/other
/privacy-policy
/privacy
etc.
I've already bulk created 301-redirects by uploading my table as a CSV into the Redirection plug-in.
But I am now looking for a way to bulk update the in-text URLs of my posts and pages so that they don't rely on the redirects, which I believe is preferable for performance and SEO. All "Search and Replace" plug-ins available only take one target and source URL at a time, which would obviously take ages.
So I believe the solution is to run an SQL script in my database. Of course, I would have to re-format my table for this, but that's fine if the script would be able to execute all 1100 search and replace commands at once. Would anyone have a suggestion?
With best wishes,
Brief
Searched what feels like the whole internet. Pretty sure there's no WP plug-in for this.
Case:
A old website developed with wordpress and run for serveral years.
Now, this wordpress website is giveup and will be re-developed to a new website which is not using wordpress anymore.
Old data i.e. article content, uploaded image file, members' info etc will be migrated to the new website.
Question:
1. Is it any tools for this type of migration?
2. If there is no tool for data transfer and I need to export the content from mysql, where can I get the article content in database?
3. Where can I find the uploaded file?
Data migration is always dependant on two things.
1. If your current framework/CMS can export the data you have, and
2. If the new framework/CMS can import data that you have just exported.
Since both frameworks are different, i.e. WordPress to any other CMS, your new CMS must be able to map the exported XML files. Which means, it must be able to customize the import process to tell what bits of data is Title, Content, Categories etc.
To answer your questions
Is it any tools for this type of migration?
Since you didn't mention what you new framework is, no one can really suggest a tool. Just google for "WordPress to YOUR-NEW-FRAMEWORK migraiton" and you'll get a number of tools. If the new framework is a custom PHP framework then I think you'll have to import the data manually or code an import script for it.
WordPress has a pretty good export tool. Check this page to learn how to export WordPress content.
Where can I find the uploaded file?
The uploaded files are referenced in the exported XML file, so at the time of import anywhere else, they can be fetched and downloaded if you can. Otherwise, you can find them in the uploads directory located at YOUR-SITE-ROOT/wp-content/uploads/
UPDATE:
As you are not targetting any framework/CMS but want to show the data with custom PHP, I'd suggest two methods.
Use the current database that WordPress is using and build your PHP script around it, benefiting from wp_posts and wp_postmeta tables.
A better way to go with this is using the WP REST API. This way, you keep the WordPress installation as is and still use it's data in your custom PHP site. Advantage of this method is that you won't have to bug your mind with scripting around database structures you have less information of.
I need to clone one of my WordPress websites with all databases and plugin settings in order to create a 1/1 copy of the whole theme, backend and functionalities except for contents. I don't want to export posts, pages, media files, comments and every other post type or so, but custom fields, posts database structure etc.
Are there any plugin to do this?
Or what queries do I have to run via SQL to do this?
Thanks
I've done many similar things, and honestly this would probably be the most time-efficient for you unless you have thousands of posts/pages or something.
Install BackUpWordPress and create a backup of all your files and the database.
Extract the sql file from that archive. Do a search/replace to swap out URLs if the domain name is changing. Then import it into your new database.
Upload the entire zip file onto your web server.
Extract it.
Manually delete all site content from the WordPress backend. Again, unless you have many thousands of posts and pages, this doesn't really take all that long. You can use screen options to display more than 10 or 20 pages/posts at a time for bulk deletion, but I wouldn't recommend going much higher than 100.
That's it! Sometimes brute force is the fastest way. Don't make things more complicated than they have to be.
I'm working on a online file library for one of my clients. It's a library for mostly PDFs and office documents. Because they are a huge amount(almost 2gb in files), I'm hosting them on another site(divshare), so that the hosting account we have doesn't get blocked by the excess of files, and also because of excess traffic downloading can generate.
So, my question is if there is a good download manager(even with some search for download categories and so) that can handle instead of local uploaded files, URLs of files hosted somewhere else?
The advice about any plugin or the like is very appreciated.
You ask how to code such manager, right? Otherwise, your are on the wrong site...
I think you need to make a kind of file manager, except that instead of generating HTML pages to view the files and act on them, it exposes a Web API, returning XML or Json data, that a WordPress plugin can manage.
From Divshare:
The DivShare Uploader Plugin for Wordpress replaces your regular
uploading frame with a DivShare upload form, allowing you to easily
upload and add files without ever leaving your "Write a Post" page.
It's a great way to speed up your blogging and take the load off your
servers when hosting big files and images.
It can be found here: http://www.divshare.com/integrate
Good Luck!
Marcelous
I need to create an RSS feed from a sqlite3 database containing blog posts. The site that will use the feed is on Google App Engine but is a static site currently. To create an RSS feed would it be easier to use a server-side script in Python or to use a Windows software tool (e.g. FeedForAll) to produce the news feed, say, once a week? Or is there a simple way to format the feed in a text editor?
Of course you can manually create your feed with a text editor, but your maintenance effort would skyrocket. (Oh, there is a new blogpost, let's edit the feed, paste in the blog post, delete an old blog post....)
Using a desktop tool to generate the feed would just automate the manual process (if you have an extremely high server load, this is a valid idea though).
In the end, you have a database from which you wish to serve textfiles to the cloud. This is something that server-side scripts like ASP.NET / PHP, etc. are meant to do and have a rather history of doing.