I have an options page in my Chrome extension with a very large list of values that get saved to chrome.storage. These options get used across several different websites. In order to use the options I have to retrieve them in my content scripts that load on these websites.
I'm trying to figure out the best way to handle this. I'm fairly confident that once I get my options loaded during a session on a website that I would want to save them to sessionstorage or localStorage so that I can have quick synchronous access to them on subsequent page loads of that same site.
But the initial retrieval of those options is where I'm wondering what's the best method. My gut says to just make all of the asynchronous calls for each individual option from my content script at document_start and return a promise once I've retrieved them all using Promise.all().
My other idea was using message passing to talk to the background page where all of my options can already be available because I will have done the async calls when Chrome started or because I could just skip chrome.storage and store them in the background pages' localStorage. In this case, each different website that needs the options is just sending one message to the background page instead of making 40+ async calls to chrome.storage.
Related
I have a Next.js app which has multiple pages. Each page has the same bar at the top of the screen which displays some data which is fetched from an internal API using SWR. The issue I am having is that to fetch that data it requires an ID of the logged in user (they log in with Discord via next-auth) and after they have left the first page, that ID value is no longer accessible to me.
I have tired storing it in local storage and session storage, I have tried passing it through other components, but nothing has worked and I am out of ideas of how to fix this. Is there any other way where I can pass a value between pages? Or is there a way I can access the session data again?
If you use this on all pages, you could just add that header into the _app.{js,tsx}(see the docs). In case you don't I suppose you create a layout component and reuse it across these pages.
In the future we might be able to use the new Routing API which allows passing adding Layout components for certain sub-routes. (Check the Layout RFC for more info).
because next js uses the react library so you can pass the data using props or you can use redux as well.
like this:
<Component propName={value}/>
I'm looking to get structured article data from webpage urls. So far I've found these two services http://www.diffbot.com/ and http://embed.ly/extract/demos/nlp. Are there better alternatives or is it worthwhile to write the code to do this myself?
If you'd like to skip the code, and are looking for a simple software for web scraping / ETL applications, I'd suggest Foxtrot. It's easy enough to use and doesn't require coding. I use it to scrape data from certain gov't websites and dump it into an Excel spreadsheet for reporting purposes.
I have done web scraping / content extract for quite some time now.
For me the best approach is to write a Chrome content extension and automate the browser with their API. This requires that you know Javascript and HTML. In one of my recent projects I use a background page with a couple of editable divs to configure the scraping session. I have some buttons on the background page to start the process. The background page loads a JS script which listens to click events of the buttons.
When one of the buttons is clicked I add a new tab for the scraping session with chrome.tab.create. The background js also defines some chrome.tabs.onUpdated.addListener to inject content scripts when the tab url contains a specific page/domain name.
The content script then does the scraping job for example selecting some elements with jquery, regular expressions etc and finally send a message with an object back to background JS using chrome.runtime.sendmessage. The background JS script listens to messages with chrome.runtime.onMessage.addListener and acts based on the content being extracted.
The extension also automates web databases by clicking for example the next page links.
I have added a timing setting to control the amount of links being clicked / tabs being opened per minute so that the access is slowed down on purpose and too much crawling is avoided.
Finally the results are being uploaded to a database with an AJAX call and inserted with a PHP page into MySQL.
When the extension runs the next time it compares the keys/links which already exist in the database with another AJAX call and ensures that only new information is being extracted.
I have also built extension like the above with Firefox but the best and easiest solution for me is a Chrome/Chromium content extension.
We have a Google Spreadsheet from which we wish to load data into our webpage.
I started by using the Google Spreadsheet APi via C# and the Google API .NET libraries to read the spreadsheet and load it into an html unsorted list.
The spreadhsheet has about 200 rows, but could have more, as it will be updated frequently. So the problem is that the users have to wait until the spreadsheed data is retrieved and parsed before they can see anything in the webpage (the page is white whilst loading).
How can I load this data in the background whilst the page loads?
I've already written my code in C# and don't much want to spend the time swapping to javascript, but I will if I have to.
Could I use the AJAX Control Toolkit to do this? I know it will load html, but can I use it to fetch google data?
What can I do here that would be fast and easy?
[Edit]
The account that hosts the google spreadsheet is inside a google domain, so it's documents can't be shared to the public as a whole - only to individuals. The C# libraries allow me to use the account's username and password to log into the account to get the spreadsheet data, and so the spreadsheet doesn't need to be shared at all. Even if I went with a javascript/ajax solution, I would yet need this functionality.
Well, this probably isn't the BEST answer, but it IS a solution. I'd like to see if y'all have a better one.
Anyway, I found this, which is an example of how to use an asp:Timer to delay the calling of a function for a certain amount of time - in my case, long enough for the page itself to load. At least this way, the user gets to see the page, and can watch the nice loading-gif until the actual content arrives.
It is an AJAXy approach that allows me to keep my c# programming without having to add any javascript.
I have recently created an application where a lot of data is loaded into objects when the application starts up, and other data as it is required. For example if the user requests the catalogue page then it will load all the top level category data into objects of type Category. This will then stay there to be used by other users (who will therefore not have to load this data into objects) and can be altered by admin if they happen to login during the same application instance. I know this is not the most efficient solution, as pointed out below, but it works and the page load, at the moment, is not too long. It is very quick if most of the required data is already loaded into objects. It is also tailored to the business' needs - unlike other techniques such as Linq-to-SQL.
The problem I am facing is when a page is requested which requires lots of data to be displayed about different types of object. For example when a catalogue page is requested which displays information on a product which can be bought, it then loads all the products and categories (as the products make reference to the category object, not just the category name).
I would like to display a loading symbol with a message whilst all this data is being loaded into objects, so the user knows its not just in a loop or anything. Is there any way to do this? I am open to using JS / jQuery if I need to.
Thanks in advance.
Regards,
Richard
PS I am working on ways to make it more efficient - such as using HashTables or HashMaps. However this is taking time as there are so many different types of item (News, Events, Catalogue Item - Range, Collection, Design, RangeCollection, CollectionDesign, RangeCollectionDesign and RangeDesign - Users, PageViews and the list goes on).
Please correct me if I'm wrong, but I do believe that Javascript is required in order to display a "loading" image... Using server-side scriping alone would typically require an entire page load after all the content loads unless you want to start messing with IFrames.
This is a job for AJAX. A common solution to your problem is to have a small page that displays a loading icon. The page has some JavaScript that makes additional HTTP requests to the server to download the rest of the page. JQuery has a "$.ajax" method that is designed to simplify this process.
I would suggest looking at the documentation to the .ajax method in the jQuery documentation. Unfortunately, it seems to be a rather delicate process to get all the scripting code right and it takes a while to learn it all.
The task is relatively straightforward:
A Drupal website displays a list of articles with thumbnails. Some visitors would like to view it without images by clicking on a button/link and have that preference saved.
e.g. http://patterntap.com/collections/index/
The problem is all visitors are anonymous and given certain traffic, page cache is enabled.
My idea was to use some simple JavaScript to set a cookie, refresh the page and depending on the cookie values (or its presence/absence) display or hide the images.
Except Drupal serves cached pages quite early and the only quick way to modify the cached version that I could find is by hacking includes/bootstrap.inc and add a custom class to the body classes then hide the images with css.
A very wrong approach, I know. But I wonder if there is a way to save different versions of a page and serve the correct version?
Edit:
need to keep the same uri
the js to show/hide the images without reload and set the cookie is already in place
hook_boot() is not really called for cached pages, so can't do it via custom module
.htaccess mods?
Edit/solution:
In the end went with Rimian's suggestion. But it is possible to accomplish the task using our own cache.inc implementation as seen in the Mobile Tools module. Specifically, by extending cache.inc and updating settings.php to include
$conf['page_cache_fastpath'] = FALSE;
$conf['cache_inc'] = 'path/to/my/module/my_module_cache.inc';
So let me get this right. You wanna hide some images on a cached page if the user chooses to?
Why don't you write some jQuery or javascript and load that into your cached page with all the rest of the document?
Then, the client/browser would decide to run your script and hide images depending on some parameters you passed along with the request to that page or in the cookie? The script gets cached and only runs when you call it.
If you were hacking the bootstrap for something like that you'd really need to be rethinking what you were doing. Crazy! :)
Also take a look at cache_get and cache_set:
http://api.drupal.org/api/drupal/includes--cache.inc/6
I'm not sure I 100% understand what you are trying to do but here are my thoughts. One of your root problems is that you are trying to access what is essentially different content at the same uri.
If this is truly what you want to do, then Rimian's suggestion of checking out chache_get and chache_set may be worthwhile.
Personally, it seems cleaner to me to have your "with thumbnails" and "without thumbnails" be accessed via different uri's. Depending on exactly what you are wanting to accomplish, a GET variable my be an even better way to go. With either of these two options you would hide or show your thumbnails at the theme layer. Pages with different paths or get variables would get cached separately.
If you want the visitor to be able to switch views without a page reload, then jQuery and a cookie would probably suite your needs. This wouldn't require a page reload and switching back and forth would be quite simple.