(Brand new learning Electron here, so I'm sure this is a basic question and I am missing something fundamental...)
How does one interact with a local database (I am using Sqlite) from an Electron application front end? I have a very basic database manager class and have no problem using it from the index.js file in my Electron application. But from the front-end (I am using Svelte, but I can probably translate solutions from other front-end frameworks), how does one interact with the database? This seems fundamental but I'm striking out trying to find a basic example.
Since everything is local it would seem it shouldn't be necessary to set up a whole API just to marshal data back and forth, but maybe it is? But if so, how does one go about telling the Electron "backend" (if that's the right term) to do something and return the results to the front end? I'm seeing something about IPC but that's not making a lot of sense right now and seems like overkill.
Here's my simple database manager class:
const sqlite3 = require("sqlite3").verbose();
class DbManager {
#db;
open() {
this.#db = new sqlite3.Database("testing.db", sqlite3.OPEN_READWRITE);
}
close() {
this.#db.close();
}
run(sql, param) {
this.#db.run(sql, param);
return this;
}
}
const manager = new DbManager();
module.exports = manager;
And I can call this and do whatever no problem from the Electron entry point index.js:
const { app, BrowserWindow, screen } = require("electron");
require("electron-reload")(__dirname);
const db = require("./src/repository/db");
const createWindow = () => {
...
};
let window = null;
app.whenReady().then(() => {
db.open();
createWindow();
});
app.on("window-all-closed", () => {
db.close();
app.quit();
});
But what to do from my component?
<script>
// this won't work, and I wouldn't expect it to, but not sure what the alternative is
const db = require("./repository/db");
let accountName;
function addAccount() {
db.run("INSERT INTO accounts (name) VALUES ($name);", { $name: accountName });
}
</script>
<main>
<form>
<label for="account_name">Account name</label>
<input id="account_name" bind:value={accountName} />
<button on:click={addAccount}>Add account</button>
</form>
</main>
If anyone is aware of a boilerplate implementation that does something similar, that would be super helpful. Obviously this is like application 101 here; I'm just not sure how to go about this yet in Electron and would appreciate someone pointing me in the right direction.
If you're absolutely 100% sure that your app won't be accessing any remote resources then you could just expose require and whatever else you may need through the preload script, just write const nodeRequire = require; window.require = nodeRequire;.
This is a fairly broad topic and requires some reading. I'll try to give you a primer and link some resources.
Electron runs on two (or more if you open multiple windows) processes - the main process and the renderer process. The main process handles things like opening new windows, starting and closing the entire app, tray icons, window visibility etc., while the renderer process is basically like your JS code in a browser. More on Electron processes.
By default, the renderer process does not have access to a Node runtime, but it is possible to let it. You can do that in two ways, with many caveats.
One way is by setting webPreferences.nodeIntegration = true when creating the BrowserWindow (note: nodeIntegration is deprecated and weird. This allows you to use all Node APIs from your frontend code, and your snippet would work. But you probably shouldn't do that because a BrowserWindow is capable of loading external URLs, and any code included on those pages would be able to execute arbitrary code on your or your users' machines.
Another way is using the preload script. The preload script runs in the renderer process but has access to a Node runtime as well as the browser's window object (the Node globals get removed from the scope before the actual front end code runs unless nodeIntegration is true). You can simply set window.require = require and essentially work with Node code in your frontend files. But you probably shouldn't do that either, even if you're careful about what you're exposing, because it's very easy to still leave a hole and allow a potential attacker to leverage some exposed API into full access, as demonstrated here. More on Electron security.
So how to do this securely? Set webPreferences.contextIsolation to true. This definitively separates the preload script context from the renderer context, as opposed to the unreliable stripping of Node APIs that nodeIntegration: false causes, so you can be almost sure that no malicious code has full access to Node.
You can then expose specific function to the frontend from the preload, through contextBridge.exposeInMainWorld. For example:
contextBridge.exposeInMainWorld('Accounts', {
addAccount: async (accountData) => {
// validate & sanitize...
const success = await db.run('...');
return success;
}
}
This securely exposes an Accounts object with the specified methods in window on the frontend. So in your component you can write:
const accountAdded = await Accounts.addAccount({id: 123, username: 'foo'});
Note that you could still expose a method like runDbCommand(command) { db.run(command) } or even evalInNode(code) { eval(code) }, which is why I said almost sure.
You don't really need to use IPC for things like working with files or databases because those APIs are available in the preload. IPC is only required if you want to manipulate windows or trigger anything else on the main process from the renderer process.
Our company produces an automation framework that is written in svelte/sapper. One feature is that developers can create custom ui widgets, currently using plain js/html/css and our client side api. These widgets are stored in the database and not on the file system.
I think it would be a big plus to allow them to create widgets as svelte components since it contains all of the markup, js and css in one location and would give them all of the benefits of svelte's reactivity.
I have gotten as far as creating an endpoint that compiles components using svelte's server API but that just seems to generate a module that is ready for rollup-plugin-svelte/sapper/babel to finish the job of producing something the browser can use.
How can I manually compile a svelte component down to the final javascript and css that sapper/svelte produces.
Ouch, tough one. Hang tight.
What you're missing actually is the "linking", that is resolving import statements in the compiled code to something the browser can use. This is the work that is typically done by the bundler (e.g. Rollup, Webpack...).
These imports can come from user (widget developer) code. For example:
import { onMount } from 'svelte'
import { readable } from 'svelte/store'
import { fade } from 'svelte/transition'
import Foo from './Foo.svelte'
Or they can be injected by the compiler, depending on the features that are used in your component. For example:
// those ones are inescapable (bellow is just an example, you'll
// get different imports depending on what the compiled component
// actually does / uses)
import {
SvelteComponent,
detach,
element,
init,
insert,
noop,
safe_not_equal,
} from 'svelte/internal'
Svelte compiles .svelte to .js and, optionally, .css, but it doesn't do anything with imports in your code. On the contrary, it adds some (but still, doesn't resolve them, it's out of its scope).
You'd need to parse the compiled code to find those imports that, raw from the compiler, probably points to paths on your file system and your node_modules directory, and rewrite them to something that makes sense for the browser -- that is, URLs...
Doesn't seem much fun, does it? (Or too much of it, depending on how you see things...) Fortunately, you're not alone with this need and we've got pretty powerful tooling dedicated precisely to this task: enter the bundler!
Solving the linking problem
One relatively straightforward approach to this problem (more to come, don't get too excited too early) is to compile your widgets, not with Svelte's compiler API, but with Rollup and the Svelte plugin.
The Svelte plugin essentially does what you were doing with the compiler API, but Rollup will also do all the hard work of rewiring imports and dependencies in order to produce a neat little package (bundle) that is consumable by the browser (i.e. that doesn't rely on your file system).
You can compile one widget (here Foo.svelte) using some Rollup config like this:
rollup.config.Foo.js
import svelte from 'rollup-plugin-svelte'
import commonjs from '#rollup/plugin-commonjs'
import resolve from '#rollup/plugin-node-resolve'
import css from 'rollup-plugin-css-only'
import { terser } from 'rollup-plugin-terser'
const production = !process.env.ROLLUP_WATCH
// include CSS in component's JS for ease of use
//
// set to true to get separate CSS for the component (but then,
// you'll need to inject it yourself at runtime somehow)
//
const emitCss = false
const cmp = 'Foo'
export default {
// our widget as input
input: `widgets/${cmp}.svelte`,
output: {
format: 'es',
file: `public/build/widgets/${cmp}.js`,
sourcemap: true,
},
// usual plugins for Svelte... customize as needed
plugins: [
svelte({
emitCss,
compilerOptions: {
dev: !production,
},
}),
emitCss && css({ output: `${cmp}.css` }),
resolve({
browser: true,
dedupe: ['svelte'],
}),
commonjs(),
production && terser(),
],
}
Nothing very extraordinary here... This is basically the config from the official Svelte template for Rollup, minus the parts pertaining to the dev server.
Use the above config with a command like this:
rollup --config rollup.config.Foo.js
And you'll get your browser-ready compiled Foo widget in public/build/Foo.js!
Rollup also has a JS API so you could run this programmatically as needed from a web server or whatever.
Then you'll be able to dynamically import and then use this module with something like this in your app:
const widget = 'Foo'
const url = `/build/widgets/${widget}.js`
const { default: WidgetComponent } = await import(url)
const cmp = new WidgetComponent({ target, props })
Dynamic imports will probably be necessary in your case, because you won't know about the widgets at the time you build your main app -- hence you will need to construct the import URLs dynamically like above at runtime. Note that the fact that the import URL is a dynamic string will prevent Rollup from trying to resolve it at bundle time. This means the import will end up as written above in the browser, and that it must be an URL (not a file path on your machine) that the browser will be able to resolve.
That's because we're consuming the compiled widget with a browser native dynamic import that we need to set output.format to es in the Rollup config. The Svelte component will be exposed with export default ... syntax, that modern browsers natively understand.
Dynamic imports are very well supported by current browsers. The notable exception is the "old" Edge (before it essentially became Chrome). If you need to support older browsers, polyfills are available (many of them actually -- e.g. dimport).
This config can be further automatized to be able to compile any widget, not just Foo. For example, like this:
rollup.config.widget.js
... // same as above essentially
// using Rollup's --configXxx feature to dynamically generate config
export default ({ configWidget: cmp }) => ({
input: `widgets/${cmp}.svelte`,
output: {
...
file: `public/build/widgets/${cmp}.js`,
},
...
})
You can then use it like this:
rollup --config rollup.config.widget.js --configTarget Bar
We're making progress, yet there remains a few caveats and hurdles to be aware of (and maybe optimize further -- your call).
Caveat: shared dependencies
The above approach should give you the compiled code for your widgets, that you can run in the browser, with no unresolved imports. Good. However, it does so by resolving all dependencies of a given widget when it is built, and bundling all these dependencies in the same file.
Said otherwise, all dependencies that are shared between multiple widgets will be duplicated for every widget, very notably the Svelte dependencies (i.e. imports from svelte or svelte/*). This is not all bad, because it gives you very standalone widgets... Unfortunately, this also add some weight to your widgets code. We're talking something like maybe 20-30 kb of JS added to each widgets that could be shared between all of them.
Also, as we will see soon, having independent copies of Svelte internals in your app has some drawbacks we need to take into consideration...
One easy way to extract common dependencies so they can be shared instead of duplicated is to bundle all your widgets in one pass. This might not be practicable for all the widgets of all your users, but maybe it can be doable at individual user level?
Anyway, here's the general idea. You would change the above Rollup configs to something like this:
rollup.config.widget-all.js
...
export default {
input: ['widgets/Foo.svelte', 'widgets/Bar.svelte', ...],
output: {
format: 'es',
dir: 'public/build/widgets',
},
...
}
We're passing an array of files, instead of just one, as the input (you would probably automatize this step by listing files in a given directory), and we're changing output.file to output.dir, since now we're gonna have several files generated at once. Those files will include common dependencies of your widgets that Rollup will have extracted, and that all your widgets will share between them for reuse.
Further perspectives
It would be possible to push even further, by extracting some shared dependencies (say, Svelte...) yourself and make them available as URLs to the browser (i.e. serve them with your web server). This way, you could rewrite those imports in your compiled code to those known URLs instead of relying on Rollup to resolve them.
This would reduce code duplication entirely, saving weight, and also this would allow to have a single version of those dependencies shared among all the widget that use them. Doing so would also relieve the need to build all widgets that share dependencies in one go at the same time, which is alluring... However, this would be pretty (!) complicated to setup, and you would actually hit diminishing returns fast.
In effect, when you're bundling a bunch of widgets together (or even just one) and let Rollup extract the dependencies, it is possible for the bundler to know what parts of the dependencies are actually needed by the consuming code and skip the rest (keep in mind: Rollup was built with tree shaking as one -- if not the one -- of its main priority, and Svelte was built by the same guy -- meaning: you can expect Svelte to be very tree shaking friendly!). On the other hand, if you extract some dependencies manually yourself: it relieves the need to bundle all consuming code at once, but you will have to expose the whole of the consumed dependencies, because you won't be able to know in advance the parts from them that will be needed.
It's a balance you need to find between what is efficient and what is practical, accounting for the added complexity of each solution to your setup. Given your use case, my own feeling is that the sweet spot is either bundling each widget entirely independently, or bundling a bunch of widgets from, say, the same user together to save some weight, as described above. Pushing harder would probably be an interesting technical challenge, but it would reap just little extra benefits, but somewhat exploding complexity...
OK so we now know how to bundle our widgets for the browser. We even have some degree of control on how to pack our widgets entirely standalone, or take on some extra infrastructure complexity to rather share dependencies between them and save some weight. Now, we've got a special dependency to consider, when we decide how we make our pretty little packets (err, bundles): that's Svelte itself...
Mind the trap: Svelte can't be duplicated
So we understand that when we're bundling a single widget with Rollup, all of its dependencies will be included in the "bundle" (just the one widget file in this case). If you bundle 2 widgets in this way and they share some dependencies, those dependencies will be duplicated in each of those bundles. In particular, you'd get 2 copies of Svelte, one in each widget. Likewise, the dependencies of your "main" app that are shared with some widgets will nonetheless be duplicated in the browser. You'll have multiple copies of the same code that will be used by those different bundles -- your app, different widgets...
However, there is something special about Svelte that you need to know: it doesn't support being duplicated. The svelte/internal module is stateful, it contains some global variables that would be duplicated if you have multiple copies of this code (see above). What this means, in practice, is that Svelte components that don't use the same copie of Svelte internals can't be used together.
For example, if you have a App.svelte component (your main app) and a Foo.svelte component (e.g. a user widget) that have been bundled independently, then you can't use Foo in App, or you'd get weird bugs.
This wouldn't work:
App.svelte
<script>
// as we've seen, in real life, this would surely be a
// dynamic import but whatever, you get the idea
import Foo from '/build/widgets/Foo.js'
</script>
<!-- NO -->
<Foo />
<!-- NO -->
<svelte:component this={Foo} />
That's also the reason why you have this dedupe: ['svelte'] option in the official Svelte template's Rollup config... This is intended to prevent bundling different copies of Svelte, which would happen if you ever used linked packages, for example.
Anyway, in your case it is kind of unescapable to end up with multiple copies of Svelte in the browser, since you're probably not wanting to rebuild your whole main app anytime a user adds or changes one of their widget... Except going to great lengths to extract, centralize, and rewrite the Svelte imports yourself; but, as I said, I don't believe this would be a reasonable and sustainable approach.
And so we're stuck.
Or are we?
The problem of duplicated Svelte copies only occurs when the conflicting components are part of the same components tree. That is, when you let Svelte create and manage the component instances, like above. The problem doesn't exist when you create and manage the component instances yourself.
...
const foo = new Foo({ target: document.querySelector('#foo') })
const bar = new Bar({ target: document.querySelector('#bar') })
Here foo and bar will be entirely independent component trees, as far as Svelte is concerned. Code like this will always work, irrelevantly of how and when (and with which Svelte version, etc.) Foo and Bar were compiled and bundled.
As I understand your use case, this is not a major hurdle. You won't be able to embed your users' widgets into your main app with something like <svelte:component />... However, nothing prevents you from creating and managing the widget instances in the right place yourself. You can create a wrapper component (in your main app) to generalize this approach. Something like this:
Widget.svelte
<script>
import { onDestroy } from 'svelte'
let component
export { component as this }
let target
let cmp
const create = () => {
cmp = new component({
target,
props: $$restProps,
})
}
const cleanup = () => {
if (!cmp) return
cmp.$destroy()
cmp = null
}
$: if (component && target) {
cleanup()
create()
}
$: if (cmp) {
cmp.$set($$restProps)
}
onDestroy(cleanup)
</script>
<div bind:this={target} />
We create a target DOM element from our main app, render an "external" component in it, pass down all the props (we're proxying reactivity), and don't forget to cleanup when our proxy component is destroyed.
The main limitation of such an approach is that Svelte context (setContext / getContext) of the app won't be visible to the proxied components.
Once again, this doesn't really seem like a problem in the widget use case -- maybe even better: do we really want the widgets to have access to every bits of the surrounding app? If really needed, you can always pass bits of context down to the widget components via props.
The above Widget proxy component would then be used like this in your main app:
<script>
import Widget from './Widget.svelte'
const widgetName = 'Foo'
let widget
import(`/build/widgets/${widgetName}.js`)
.then(module => {
widget = module.default
})
.catch(err => {
console.error(`Failed to load ${widgetName}`, err)
})
</script>
{#if widget}
<Widget this={widget} prop="Foo" otherProp="Bar" />
{/if}
And... Here we are? Let's sum it up!
Summary
Compile your widgets with Rollup, not Svelte compiler directly, to produce browser ready bundles.
Find the right balance between simplicity, duplication and extra weight.
Use dynamic imports to consume your widgets, that will be built independently of your main app, in the browser.
Do not try to mix together components that don't use the same copy of Svelte (essentially means bundled together, except if you've launched into some extraordinary hack). It might looks like it works at first, but it won't.
Thanks to the detailed post by #rixo I was able to get this working. I basically created a rollup.widget.js like this:
import json from '#rollup/plugin-json';
import resolve from '#rollup/plugin-node-resolve';
import commonjs from '#rollup/plugin-commonjs';
import svelte from 'rollup-plugin-svelte';
import path from "path";
import fs from "fs";
let basePath = path.join(__dirname,'../widgets');
let srcFiles = fs.readdirSync(basePath).filter(f=>path.extname(f) === '.svelte').map(m=> path.join(basePath,m ));
export default {
input: srcFiles,
output: {
format: 'es',
dir: basePath,
sourcemap: true,
},
plugins: [
json(),
svelte({
emitCss: false,
compilerOptions: {
dev: false,
},
}),
resolve({
browser: true,
dedupe: ['svelte']
}),
commonjs()
]
}
Then generate the svelte components from the database and compile:
const loadConfigFile = require('rollup/dist/loadConfigFile');
function compile(widgets){
return new Promise(function(resolve, reject){
let basePath = path.join(__dirname,'../widgets');
if (!fs.existsSync(basePath)){
fs.mkdirSync(basePath);
}
for (let w of widgets){
if (w.config.source){
let srcFile = path.join(basePath,w.name + '.svelte');
fs.writeFileSync(srcFile,w.config.source);
console.log('writing widget source file:', srcFile)
}
}
//ripped off directly from the rollup docs
loadConfigFile(path.resolve(__dirname, 'rollup.widgets.js'), { format: 'es' }).then(
async ({ options, warnings }) => {
console.log(`widget warning count: ${warnings.count}`);
warnings.flush();
for (const optionsObj of options) {
const bundle = await rollup(optionsObj);
await Promise.all(optionsObj.output.map(bundle.write));
}
resolve({success: true});
}
).catch(function(x){
reject(x);
})
})
}
And then consume the dynamic widget as #rixo proposed:
<script>
import {onMount, onDestroy, tick} from 'svelte';
import Widget from "../containers/Widget.svelte";
export let title = '';
export let name = '';
export let config = {};
let component;
let target;
$: if (name){
loadComponent().then(f=>{}).catch(x=> console.warn(x.message));
}
onMount(async function () {
console.log('svelte widget mounted');
})
onDestroy(cleanup);
async function cleanup(){
if (component){
console.log('cleaning up svelte widget');
component.$destroy();
component = null;
await tick();
}
}
async function loadComponent(){
await cleanup();
let url = `/widgets/${name}.js?${parseInt(Math.random() * 1000000)}`
let comp = await import(url);
component = new comp.default({
target: target,
props: config.props || {}
})
console.log('loading svelte widget component:', url);
}
</script>
<Widget name={name} title={title} {...config}>
<div bind:this={target} class="svelte-widget-wrapper"></div>
</Widget>
A few notes/observations:
I had much better luck using rollup/dist/loadConfigFile than trying to use rollup.rollup directly.
I went down a rabbit hole of trying to create both client and server globals for all of the svelte modules and marking them as external in the widget rollup so that everything used the same svelte internals. This ended up being a mess and gave the widgets access to more than I wanted.
If you try to embed your dynamically compiled widget in your main app with <svelte:component it will sort of work but give you the dreaded outros.c undefined error if you try to reference a dynamic widget from another. After this happens reality breaks down and the app is in a strange state.
#rixo is always right. I was warned about each of these things in advance and the result was exactly as predicted.
I'm working on a Symfony application and just got SSR for JS working using https://github.com/spatie/server-side-rendering. So far I only worked with "readymade" SSR solutions for React, but currently I'm trying to use hyperHTML/viperHTML and am facing a few issues that so far I wasn't able to solve by looking at the available docs/examples.
My current test snippet is this:
const viperHTML = require('viperhtml');
class Component extends viperHTML.Component {
constructor(props) {
super();
this.props = props;
}
render() {
return this.html`
<h1>Hello, ${this.props.name}</h1>`;
}
}
console.log(new Component({ name: 'Joe' }).render().toString());
The thing here is that without explicitly calling render() I get no output. Looking at some of the official examples this shouldn't be necessary, at least not with Component. I already tried using setState() in the constructor, for example, but no difference.
Also, without using both, console.log() and toString(), I get no output either. Which is unexpected. I get that toString() might be necessary here (without it a <buffer /> is being rendered), but the console.log() seems odd. This might not be related to viperHTML at all of course. But instantiating the component is the only thing I expected to be necessary.
It's also not clear to me yet how I can write an isomorphic/universal component, i.e. one file which has the markup, event handlers etc., gets rendered on the server and then hydrated on the client. When I add an inline event handler as per the docs (https://viperhtml.js.org/hyperhtml/documentation/#essentials-6) it actually gets inlined into the rendered markup, which is not what I want.
I checked hypermorphic and the viperNews app, but that didn't really help me so far.
In case it helps, you can read viperHTML tests to see how components can be used.
The thing here is that without explicitly calling render() I get no output.
Components are meant to be used to render layout, either on the server or on the client side. This means if you pass a component instance to a hyper/viperHTML view, you don't have to worry about calling anything, it's done for you.
const {bind, Component} = require('viperhtml');
class Hello extends Component {
constructor(props) {
super().props = props;
}
render() {
return this.html`<h1>Hello, ${this.props.name}</h1>`;
}
}
console.log(
// you need a hyper/viperHTML literal to render components
bind({any:'ref'})`${Hello.for({ name: 'Joe' })}`
// by default you have a buffer to stream in NodeJS
// if you want a string you need to use toString()
.toString()
);
Since NodeJS by default streams buffers, any layout produced by viperHTML will be buffers and, as such, can be streamed while it's composed (i.e. with Promises as interpolation values).
It's also not clear to me yet how I can write an isomorphic/universal component, i.e. one file which has the markup, event handlers etc., gets rendered on the server and then hydrated on the client.
The original version of hyperHTML had a method called adopt() which purpose was to hydrate live nodes through same template literals.
While viperHTML has an viperhtml.adoptable = true switch to render adoptable content, hyperHTML adopt feature is still not quite there yet so that, for the time being, you can easily share views between SSR and the FE, but you need to either take over on the client once the SSR page has landed or react, for the very first time, differently and take over on the client at distance.
This is not optimal, but I'm afraid the hydration bit, done right, is time consuming and I haven't found such time to finalize it and ship it.
That might be hyperHTML v3 at this point.
I hope this answer helped understanding how viperHTML works and what's the current status.
In Selenium, it's possible to extend elements. This makes it possible to have a set of reusable custom elements for testing.
For example, we can have a getText method added.
public static string GetText(this IWebDriver driver)
{
return driver.FindElement(By.TagName("body")).Text;
}
And re-use it as follows:
myElement.getText();
This example is detailed here: http://www.vcskicks.com/selenium-extensions.php
Is there a way to replicate this behavior in Cypress.io? Or do we need to query and call the same methods to get the data?
You can accomplish this using simple functions. For example, you could move several functions into a utils.js.
export const getByText = (text) => cy.contains(text)
And then import those methods into your spec file.
import { getByText } from './utils'
it('finds the element', () => {
getByText('Jane Lane')
})
You could alternatively create a custom command, but that's sometimes not necessary as noted in the best practices.
I think that Custom Commands is what you're looking for. However, note the Cypress Best Practices. The basic commands are very powerful and you can accomplish a lot with them.