I've been looking at Jupyter Lab's GitHub repository (issues, roadmap, etc.), and I couldn't find any indices regarding the progress toward resistance to network issues or browser reload.
I first noticed this with Z2JH v2.0, which I use at work. The same issue is with the latest version of JupyterLab (v3.4.8) installed with pip on my machine. Running the sample below and reloading the browser in the middle of execution would end up at the last saved state, and the cell's output would not continue updating. Any other changes after saving are also lost.
import time
def long_running_task():
for i in range(10):
time.sleep(1)
print(f'Done #{i+1}!')
long_running_task()
I came across several hacks/workarounds, such as writing to file and IPython magic %%capture. However, running a long-running task becomes frustrating when I need to move around, close the laptop and migrate to a different location.
Alternatives? Suggestions?
I am evaluating a self-hosted artifactory installation on a trial license. I followed the official installation instructions for the docker container and the linux archive file. Neither of these installation options are working. The artifactory service fails to start.
I have opened an issue to track the problem: https://www.jfrog.com/jira/browse/RTFACT-27182
TL;DR; A component fails, a nasty stack trace appears in the logs, and eventually the services stop.
It would seem that there is a bug in artifactory. I have traced this back to multiple versions and this issue spans multiple years.
The problem appears to be that artifactory cannot get past the bootstrapping/initialization phase when started with artifactoryctl. At a certain point (around 2-5 minutes in) all the services stop and a pid file is left over, which is bad.
The workaround I have found is that the service can pass this initialization phase only after multiple start/stops (3 to be exact). In other words, we call artifactoryctl start, wait for all failures, then artifactoryctl stop and repeat two more times. On the fourth and final start, we will see the service come online (in about 150 - 190s). From then on, the service will start correctly with one call to artifactoryctl start.
I have not yet looked at the systemd unit file. My guess would be that it has/or could be made to have a number of retries to work around this issue and perhapse this issue does not exist when using the service wrapper.
I have also not yet looked again at the docker container which appears to be failing for the same reason. A workaround off the top of my head would be to modify the entrypoint script. If you were to dockerk exec into the container and try the workaround above it would likely terminate the root process and kill the container.
Just recently I am starting to experience an issue when trying to deploy my functions through Firebase. When running the command firebase deploy --only functions one of my four functions deploy correctly, but the other three throw this error in the firebase-tools console:
Deployment error.
Build failed: Build error details not available. Please check the logs at url to my GCP logs
When I look at the logs the error message that shows for each function that is failing is:
ERROR: build step 3 "us.gcr.io/fn-img/buildpacks/nodejs10/builder:nodejs10_20200922_20_RC00" failed: step exited with non-zero status: 46
I have searched and googled for quite a while now and I cannot find anyone reporting the same error code, having the same non-zero status returned, or having functions fail on step 3. There doesn't seem to be much for documentation on troubleshooting the errors either.
Hopefully someone here can point me in the right direction.
Thanks!
The error seems to be caused by a missing (or not accessible) file in the history/restore functionality of Firebase. Don't know the reason for this.
A workaround that worked for me was:
Remove the functions in the web console
Deploy normally > firebase deploy
Answering since I can't comment in response to xaphod, Markus's fix worked for me, thank you mate! Deleting the function means it creates a new one rather than trying to restore it, which is the failed step 3 in the build process.
In response to xaphod, I did setup a lifecycle rule for old files in the artifacts bucket, it was by far my biggest bucket in storage and was pushing the free tier limit. I set it up a bit short probably, delete 5 days after last updated time, but surprisingly didn't run into this issue until today (possibly because I usually just deploy them all whenever I update one). I removed the lifecycle rule for now, although I'd rather it not run up the storage space again, I might end up contacting support to see if there's a way to lifecycle them without deleting the most recent, not sure why it doesn't just clear out the old images on updates though.
Found the post that suggested setting up lifecycle rules, firebase storage artifacts is huge, and did some testing as well, deleted all the files in us.artifacts and deployed all my functions (got errors on all them, only deleted the function that I needed to update earlier and deployed it by itself). Doesn't cause any errors, just takes a bit longer, so I think the issue can be fixed by deleting them all or deleting nothing, probably going to set the lifecycle rule to delete immediately and deal with the increased deploy time, rather than the cost of firebase storage.
Update: The problems passed for a while, but then returned with a vengeance since yesterday. Deploys now really take forever and always fail with Server Error. connect ETIMEDOUT or Upload Error: Cannot read property 'response' of undefined or something else.
After experimenting with connecting via a US location using HideMyAss I found that that completely resolved my issues though! Note that this issue not only occurs when deploying from our office in Amsterdam, but also from our office in Rotterdam. In the meanwhile I have also heard from more people experiencing issues with other Google services.
I have replied to the related Firebase Support email with this information and hope they will look into it. In the meanwhile I guess I'll have to keep on using HideMyAss..
--
Deploying Functions has been taking increasingly longer times after adding more of them. Occasionally at first, but recently for some periods every time I try to $ firebase deploy --only functions one of the functions being deployed at random fails with:
⚠ functions[foo]: Deploy Error: Failure in the
execution environment
When I try again an hour or so later it deploys without a problem (still takes 2 minutes to deploy which seems a little slow).
Perhaps the deploy process is timing out; it always fails after a long time, never quickly.
Perhaps my location outside of America is resulting in latency related issues in the deployment process? Doesn't seem very likely though..
I'm also looking into Firebase Functions logging "Function execution took 60002 ms, finished with status: 'timeout'" and other performance issues, so I wonder if these are all related.
PS: I also reported this to https://firebase.google.com/support/ but the last report I made there is still un-answered after 15 days, so I'm going to go ahead and post it here as well. I included a firebase-debug.log with that report, but rather not publicly sharing that here (not sure if there any tokens in there etc).
Also having issues
I have been having similar issues today and I am deploying from the United States.
For example a function will fail showing these two errors:
Deploy Error: Failure in the execution environment
Error: Functions did not deploy properly.
I found that once when this happened it was because my internet connection was dropping and then totally dropped during a firebase deploy.
Then again it happened because I was trying to deploy at the exact same time that at lot of my cloud functions happened to kick off doing stuff on their servers.
Once the cloud function had failed, then even when my internet resumed and my functions weren't busily running, it would not let me redeploy to get the jammed function running again. No matter what the broken copy of it was stuck in their servers with the tag:
Failure in execution environment
My Solution
I found that once you have a problem like that you can actually rename the function. On the next deploy it will effectively delete the old broken function and reload it as a working copy with the new name.
I would think if your project required it to have the same name you could do this process one more time, deleting the copy with the new name and restoring the copy with the name you needed. Or even block commenting out the function to delete it on the first deploy and then uncommenting it on another deploy to reinstall it.
How that helps you
I'm hoping that if you are still having issues from the Netherlands it is helpful to know that it may be a slow connection issue or a busy server issue. As I have found these two things to cause problems for me from within the States.
Also my solution of deleting and redeploying the function might help to speed up a deploy if the issue is with a copy that is on their servers. It would be interesting to know if that helps because even though the function looks ok on the server maybe it had issues during the previous deploy that are jamming up future deploys.
Sorry for the late response to this, hopefully you are no longer having these issues with firebase (I hate firebase btw, always issues like this).
Assume that I set up an automatic nightly build. What artifacts of the build should I save?
For example:
Input source code
output binaries
Also, how long should I save them, and where?
Do your answers change if I do Continuous Integration?
You shouldn't save anything for the sake of saving it. you should save it because you need it (i.e., QA uses nightly builds to test). At which point, "how long to save it" becomes however long QA wants them.
i wouldn't "save" source code so much as tag/label it. I don't know what source control you're using, but tagging is trivial (performance & disk space) for any quality source control system. Once your build is tagged, unless you need binaries, there really isn't any benefit to just having them around because you can simply re-compile when necessary from source.
Most CI tools let you tag on each successful build. This can become problematic for some systems as you can easily have 100+ tags a day. For such cases I recommend still running a nightly build and only tagging that.
Here are some artifacts/information that I'm used to keep at each build:
The tag name of the snapshot you are building (tag and do a clean checkout before you build)
The build scripts themselfs or their version number (if you treat them as a separate project with its own version control)
The output of the build script: logs and final product
A snapshot of your environment:
compiler version
build tool version
libraries and dll/libs versions
database version (client & server)
ide version
script interpreter version
OS version
source control version (client and server)
versions of other tools used in the process and everything else that might influence the content of your build products. I usually do this with a script that queries all this information and logs it to a text file that should be stored with the other build artifacts.
Ask yourself this question: "if something destroys entirely my build/development environment what information would I need to create a new one so I can redo my build #6547 and end up with the exact same result I got the first time?"
Your answer is what you should keep at each build and it will be a subset or superset of the things I already mentioned.
You can store everything in your SCM (I'd recommend a separate repository), but in this case your question on how long you should keep the items looses sense. Or you should store it to zipped folders or burn a cd/dvd with the build result and artifacts. Whatever you choose, have a backup copy.
You should store them as long as you might need them. How long, will depend on your development team pace and your release cycle.
And no, I don't think it changes if you do continous integration.
This isn't a direct answer to your question, but don't forget to version control the nightly build setup itself. When the project structure changes, you may have to change the build process, which will break older builds from that point on.
In addition to the binaries as everyone else has mentioned I would recomend setting up a symbol server and a source server and making sure you get the correct information out and into those. It will aid in debugging tremendously.
We save the binaries, stripped and unstripped (so we have the exactly same binary, once with and once without debug symbols). Further we build everything twice, once with debug output enabled and once without (again, stripped and unstripped, so every build result in 4 binaries). The build is stored to a directory according to SVN revision number. That way we can always retain the source from the SVN repository by simply checking out this very revision (that way the source is archived as well).
A surprising one I learned about recently: If you're in an environment that might be audited you'll want to save all the output of your build, the script output, the compiler output, etc.
That's the only way you can verify your compiler settings, build steps, etc.
Also, how long to save them for, and where to save them?
Save them until you know that build won't be going to production, iow as long as you have the compiled bits around.
One logical place to save them is your SCM system. Another option is to use a tool that will automatically save them for you, like AnthillPro and its ilk.
We're doing something close to "embedded" development here, and I can tell you what we save:
the SVN revision number and timestamp, as well as the machine it was built on and by whom (also burned into the build binaries)
a full build log, showing whether it was a full/incremental build, any interesting (STDERR) output the data baking tools produced, a list of files compiled and any compiler warnings (this compresses very well, being text)
the actual binaries (for anywhere from 1-8 build configurations)
files produced as a side effect of linking: a linker command file, address map, and a sort of "manifest" file indicating what was burned into the final binaries (CRC and size for each), as well as the debugging database (.pdb equivalent)
We also mail out the result of running some tools over the "side-effect" files to interested users. We don't actually archive these since we can reproduce them later, but these reports include:
total and delta of filesystem size, broken down by file type and/or directory
total and delta of code section sizes (.text, .data, .rodata, .bss, .sinit, etc)
When we have unit tests or functional tests (e.g. smoke tests) running, those results show up in the build log.
We've not thrown out anything yet -- given, our target builds usually end up at ~16 or 32 MiB per configuration, and they're fairly compressible.
We do keep uncompressed copies of the binaries around for 1 week for ease of access; after that we keep only the lightly compressed version. About once a month we have a script that extracts each .zip that the build process produces and 7-zips a whole month of build outputs together (which takes advantage of only having small differences per build).
An average day might have a dozen or two builds per project... The buildserver wakes up about every 5 minutes to check for relevant differences and builds. A full .7z on a large very active project for one month might be 7-10GiB, but it's certainly affordable.
For the most part, we've been able to diagnose everything this way. Occasionally there's a hiccup on the buildsystem and a file isn't actually a the revision it's supposed to be when a build happens, but there's usually enough evidence of this in the logs. Sometimes we have to dig out a tool that understands the debugging database format and feed it a few addresses to diagnose a crash (we have automatic stackdumps built into the product). But usually all the information needed is there.
We haven't had to crack the .7z archives yet, to mention. But we have the info there, and I have some interesting ideas on how to mine bits of useful data from it.
Save what can't be reproduced easily. I work on FPGAs where only the FPGA team have the tools and some cores (libraries) of the design are licensed to compile on only one machine. So we save the output bitstreams. But try to check them over one another rather than with a date/time/version stamp.
Save as in check in to source code control or just on disk? Save nothing to source code control. All derived files should be visible in the file system and available to developers. Don't checkin binaries, code generated from XML files, message digests etc. A separate packaging step will make these end products available. As you have the change number you can always reproduce the build if necessary assuming of course everything you need to do a build is completely in the tree and is available to all builds by syncing.
I would save your built binaries for exactly as long as they have a chance to go into production or be used by some other team (like a QA group). Once something has left production, what you do with it can vary a lot. For a lot of teams, they'll keep just their most recent prior build around (for rollback) and otherwise discard their builds.
Others have regulatory requirements to keep anything that went into production around for as long as seven years (banks). If you are a product company, I'd keep around any binary a customer might have installed in case a tech support guy wants to install the same version.