Cygwin SVN: E200030: SQLite disk I/O error - sqlite

When I use Subversion in Cygwin to update some repository, some directories update with success, while some other one gets a failure with the error message:
svn: E200030: sqlite: disk I/O error
When doing svn update again for the same repository, a different directory can get the same error. Sometimes, there is a SVN instruction after the above error message.

This happened due to a change someone wanted in Cygwin's SQLite package. I was the maintainer of that package when this question was asked, and I made the change that caused this symptom.
The change was released as Cygwin SQLite version 3.7.12.1-1, and it fixed that one person's problem, but it had this bad side effect of preventing Cygwin's Subversion package from cooperating with native Windows Subversion implementations.
What Happened?
The core issue here is that Subversion 1.7 changed the working copy on-disk format. Part of that change involves a new SQLite database file, .svn/wc.db. Now, in order to implement SQLite's concurrency guarantees, SQLite locks the database file while it is accessing it.
That's all fine and sensible, but you run into a problem when you try to mix Windows native and POSIX file locking semantics. On Windows, file locking almost always means mandatory locking, but on Linux systems — which Cygwin is trying to emulate — locking usually means advisory locking instead.
That helps understand where the "disk I/O error" comes from.
The Cygwin SQLite 3.7.12.1-1 change was to build the library in "Unix mode" instead of "Cygwin mode." In Cygwin mode, the library uses Windows native file locking, which goes against the philosophy of Cygwin: where possible, Cygwin packages call POSIX functions instead of direct to the Windows API, so that cygwin1.dll can provide the proper POSIX semantics.
POSIX advisory file locking is exactly what you want with SQLite when all the programs accessing the SQLite DBs in question are built with Cygwin, which is the default assumption within Cygwin. But, when you run a Windows native Subversion program like TortoiseSVN alongside a pure POSIX Cygwin svn, you get a conflict. When the TortoiseSVN Windows Explorer shell extension has the .svn/wc.db file locked with a mandatory lock and Cygwin svn comes along and tries an advisory lock on it, it fails immediately. Cygwin svn assumes a lock attempt will either succeed immediately or block until it can succeed, so it incorrectly interprets the lock failure as a disk I/O error.
How Did We Solve This Dilemma?
Within Cygwin, we always try to play nice with Windows native programs where possible. The trick was to find a way to do that, while still playing nice with Cygwin programs, too.
Not everyone agreed that we should attempt this. "Cygwin SQLite is part of Cygwin, so it only needs to work well with other Cygwin programs," one group would say. The counterpartisans would reply, "Cygwin runs on Windows, so it has to perform well with other Windows programs."
Fortunately, we came up with a way to make both groups happy.
As part of the Cygwin SQLite 3.7.17-x packaging effort, I tested a new feature that Corinna Vinschen added to cygwin1.dll version 1.7.19. It allowed a program to request mandatory file locking through the BSD file locking APIs. My part of the change was to make Cygwin SQLite turn this feature on and off at the user's direction, allowing the same package to meet the needs of both the Cygwin-centric and Windows-native camps.
This Cygwin DLL feature was further improved in 1.7.20, and I released Cygwin SQLite 3.7.13-3 using the finalized locking semantics. This version allowed a choice of three locking strategies: POSIX advisory locking, BSD advisory locking, and BSD/Cygwin mandatory locking. So far, the latter strategy has proven to be completely compatible with native Windows locking.
Later, when Jan Nijtmans took over maintenance of Cygwin SQLite, he further enhanced this mechanism by fully integrating it with the SQLite VFS layer. This allowed a fourth option: the native Windows locking that Cygwin SQLite used to use before we started on this journey. This is mostly a hedge against the possibility that the BSD/Windows locking strategy doesn't cooperate cleanly with a native Windows SQLite program. So far as I know, no one has ever needed to use this option, but it's nice to know it's there.
Alternate Remedy
If the conflict you're having is between Cygwin's command line svn and the TortoiseSVN Windows Explorer shell extension, there's another option to fix it. TortoiseSVN ships with native Windows Subversion command-line programs as well. If you put these in your PATH ahead of Cygwin's bin directory, you shouldn't run into this problem at all.

Having encountered the same problem, it appears (in my case at least) to be an interaction with TortoiseSVN. Disabling TortoiseSVN's status icon cache (Settings > Icon Overlays > Status cache "None" > Apply) has everything working just fine for me.
(That obviously doesn't resolve the underlying problem, which appears to be due to the SQL package that Cygwin's Subversion package relies on changing its mode of access. As I write, there's active [if slow] discussion on the Cygwin mailing list about how to resolve this.)

ldd /usr/bin/svn shows that SVN depends on /usr/bin/cygsqlite3-0.dll.
After I change libsqlite3 from 3.7.12 back to 3.7.3, the problem seems to go away. So this may be a SQLite library problem.

Using TortoiseSVN, ticking off Refresh shell overlays at clean up solved the problem for me.

For others reference, I just had this same error (svn: E200030: sqlite: disk I/O error) and found that one of my log files was taking up all my space (and could not write to the HDD because there was no free space).
Run (to make sure you have enough disk space)
df -h
(If you don't delete some large files (I just removed some backup and log files)
Then I just needed to run:
svn cleanup
This resolved the error for me.

Related

Where can I download older SQLite binaries?

I've got a problem with my web application. A query that completes in a few seconds in SQLite 3.7.4 on Windows takes a whole 6 minutes in SQLite 3.6.22 on Linux. On the very same database file. It contains a few WHERE NOT EXISTS parts so I can imagine something has been optimised there.
But the SQLite version is fixed on the Linux side, it's bound to the distribution so I cannot update it. When I tried a current Linux binary and uploaded it to the server, the shell just said the file would not exist. It was there, ls found it and it was mode 755 by my own user...
And I can't find that specific version for Windows as well to try it in the other direction. The SQLite download site only contains the very current version and nothing else. There's not even an archive of older releases...
Does anybody know where I can find older versions? Preferably as binary as I don't want to spent hours to get it compiled.
You can check out and compile older releases from the Fossil repository.

Running R from CD-ROM

Can R be run from a CD-ROM drive? The computer is a stand-alone (no network or Internet connection) and I can't install anything on it, nor can I use a flash drive.
Thanks.
What do you mean by "can't install"?
You don't need to install R, you can just run it from a folder copied from somewhere else. If you have hard disk storage on the PC then you can copy C:\Program Files\R from one machine onto a CD-ROM, then take the CD-ROM to the cripplebox, copy it to wherever you store your files and run it from there. Worst case scenario is you have to change the R_HOME environment variable. Works for Linux and Windows (you didnt say what OS you are on).
...unless your sysadmins have disabled executable permissions for your hard disk storage. Which is a real BOFH thing to do.
...but if they've done that I'd also suspect they've disabled executables from CD-ROM too.
...and if you don't have any writable hard disk storage, how the heck are you going to do any analysis?
...the real fix may be to kick the sysadmins until you tell them you can't do your job without R installed on the machine.
You may have trouble with packages, but otherwise, the instructions for installing R on a USB key should be pertinent.

How can I enable auto-updates in a Qt cross-platform application?

I love applications that are able to update themselves without any effort from the user (think: Sparkle framework for Mac). Is there any code/library I can leverage to do this in a Qt application, without having to worry about the OS details?
At least for Windows, Mac and user-owned Linux binaries.
I could integrate Sparkle on the Mac version, code something for the Linux case (only for a standalone, user-owned binary; I won't mess with distribution packaging, if my program is ever packaged), and find someone to help me on the Windows side, but that's horribly painful.
It is not a complete solution, but a cross-platform (Windows, Mac, Linux) tool for creating packages for auto-updates and installing them is available at https://github.com/mendeley/Update-Installer. This tool does not deal with publishing updates or downloading them.
This was written for use with a Qt-based application but to make the update installer small, standalone and easy to build, the installer uses only standard system libraries (C++ runtime, pthreads/libz/libbz2 on Linux/Mac, Win32 API on Windows, Cocoa on Mac, GTK with fallback on Linux). This simplifies delivering updates which include new versions of Qt and other non-system libraries that your application may depend on.
Before considering this though, I would suggest:
If you are only building for two platforms, consider using standard and well-tested auto-update frameworks for those platforms - eg. Sparkle on Mac, Google's Omaha on Windows or auto-update systems built into popular install frameworks (eg. InstallShield). I haven't tried BitRock.
On Mac, the Mac App Store may be a good option. See https://bugreports.qt.io/browse/QTBUG-16549 though.
On Linux, consider creating a .deb package and a simple repository to host it. Once users have a repository set up, the system-wide software update tools will take care of checking for and installing new releases. The steps for setting up a new repository however are too complex for many new Ubuntu/Debian users. What we did, and also what Dropbox and Google have done, is to create a .deb package which sets up the repository as part of the package installation.
A few other notes on creating an updater:
On Windows Vista/7, if the application is installed system-wide (eg. in C:\Program Files\$APPNAME) your users will see a scary UAC prompt when the updater tries to obtain permissions to write to the install directory. This can be avoided either by installing to a user-writable directory (I gather that this is what Google Chrome does) or by obtaining an Authenticode certificate and using it to sign the updater binary.
On Windows Vista/7, an application .exe or DLL cannot be deleted if in use, but the updater can move the existing .exe/DLL out of the way into a temporary directory and schedule it for deletion on the next reboot.
On Ubuntu, 3rd-party repositories are disabled after distribution updates. Google works around this by creating a cron-job to re-add the repository if necessary.
Shameless plug: Fervor, a simple multiplatform (Qt-based) application autoupdater inspired by Sparkle.
Shameless plug: this a relatively old question, but I thought that it may be useful to mention a library that I created recently, which I named "QSimpleUpdater". Aside from notifying you if there's a newer version, it allows you to download the change log in any format (such as HTML or RTF) and download the updates directly from your application using a dialog.
As you may expect from a Qt project, it works on any platform supported by Qt (tested on Windows, Mac & Linux).
Links:
Website
GitHub repository
Screenshot:
Though it works a bit differently than Sparkle, BitRock InstallBuilder contains an autoupdater written in Qt that can be used independently (disclaimer, I am the original BitRock developer). It is a commercial app, but we have free licenses for open source projects.
I've developed an auto-updater library which works beautifully on Mac OS X, Linux and pretty much every Unix that allows you to unlink a file while the file is still open. The reason being that I simply extracted the downloaded package on top of the existing application. Unfortunately, because I relied on this functionality, I ran into problems on Windows as Windows does not let you unlink an open file.
The only alternative I could find is to use MoveFileEx with the replace on reboot flag, but that is awful.
However, renaming the working directory of the application works on Windows 7 and Windows XP. I haven't tried Windows Vista yet.
I have found WebUpdate to be quite useful, though it's written with the wxWidgets. But don't worry, it's a separate app which handles your updates. The steps to integrate it are pretty simple - just write two XML files and run the updater. And yes, it's cross-platform.
The advantage of it is it will automatically download and unzip/install all you required and not just provide a popup with a notification about a new version and a link to download it. Another thing you can do with it is customizable actions.
Project's main page is here, you can read the docs or take a look at the official tutorial.
The blog post Mixing Cocoa and Qt may solve the problem for the Mac platform.
You can use UpdateNode which gives you all the possibilities to update your software. It's using a cross platform Qt client and is free for Open Source!
UPDATE
Just did some further analysis on that and really like this solution:
Pros:
Free for Open Source!!! Even the client is Open Source: https://github.com/updatenode/unclient
The client is already localized in several languages
Very flexible in terms of updates. You can even update single non-binaries.
Provides additionally a way to display messages though the client.
Ready to use binaries & installer for all common Linux distributions, single Windows binary, as well as installer and a solution for Mac (which I have not tried, as I don't have a Mac)
Easy to use web service, nice statistics and update check is integrated within few minutes
Cons:
I am missing a multi-user management in the online service. Maybe they will do it in future - I will definitely suggest that in their feedback portal
The client is a GUI client only - so, you will need to shrink it down to run without a GUI frontend (maybe only necessary for people like me ;-) )
So, bottom line, as this solution is quite new, I think there is lot of potential here. I will definitely use it in my project and I am looking forward for more from them! Thumbs up!
This is an old question but there is not Squirrel in answers which is BEST SOLUTION , here is what I'm doing in qt 5.12.4 with qt quick "my qml app" you can do this in any other language
I'm doing this in windows there is mac version of squirrel too, I don't know about Linux
download nuget package explorer release
https://github.com/NuGetPackageExplorer/NuGetPackageExplorer/releases
open nuget package explorer and add this directory 'lib/net45' it doesn't matter you have a .net app or not, I did this for my qt application otherwise it won't work.
add all files into this folder specify your version in the metadata
save nupkg file
download squirrel release https://github.com/Squirrel/Squirrel.Windows/releases
add squirrel to windows environment path
open cmd and cd to directory of nupkg file
squirrel --releasify file_name.nupkg -> now inide releases folder, there should be setup.exe file which will install app and other files.
to create new version do 2,3,4,7,8 again if its an update it will create delta file which is only needed file to update, put this files into your service directory for example in updates folder of your website which you need to disable directory browsing in IIS , and to auto-update application you need to call Update.exe which is in parent folder of application root directory appdir/../update.exe --update http://yourserver.com/upates/ after application restart app should start with new version
you can find documentation for squirrel in https://github.com/Squirrel/Squirrel.Windows/blob/develop/docs/getting-started/0-overview.md and nuget package explorer here https://github.com/NuGetPackageExplorer/NuGetPackageExplorer and you can use only nuget.exe too if you don't want to use nuget package explorer which can be used for dynamic generation of versions, which can be download from https://www.nuget.org/downloads
That easy. Now you have auto-update app which will download updates from the server and auto-update app. For more info you can read documentations.
note: for iis uses https://github.com/Squirrel/OldSquirrelForWindows/issues/205
I suggest you read on plugin and how to create and use them. If your application architecture is modular and be split into different plugins. Take a look at Google Auto Update utility http://code.google.com/p/omaha/. We use this.
Thibault Cuvelier is writing a tutorial (in French) to develop an updater. I know the explanations are in French (and everyone is not understanding French), but I think this can be readable with a web translator like Google Translate. With this you will have a cross-platform updater, but you need to write it by yourself.
For what I know, the only part of the updater that is explained in the tutorial, is the file downloading part. In the case this can help you, refer to the tutorial, Un updater avec Qt.
I hope that helps.
OK, so I guess I take it as a "no (cross-platform) way". It's too bad!
I have found a solution that can be automated with built-in self-extracting patches and updates. for windows. I have started using their sdk. take a look at the massive documentation here, https://agersoftware.com/docs/ the sdk is called securesdk and comes with their app, SecureDelta sdk. does a great job on any kind of files, better results than lzma-included delta updaters

"Database is locked" error in SQLite over a Mac network

I have created a simple database using SQLite (actually PySQLite). It works fine when I'm querying or writing to the database from the local machine (ie program and database file on the windows machine drive). However when I copy the database file to my network drive (a time capsule), then Windows machines, although they can see the files and have full read/write access to the drive, give me a "SQL Error: database is locked" even when performing a simple select!
Queries work fine over the network from Macs.
There is no fancy multi-access going on - only one machine has the database open. Seems like some weird Mac networking issue. Happens in either the Python program, or in the SQLite3 command line. I am using SQLite 3.6.14.2.
Anybody seen this problem? Any way of fixing it? Don't really want to get heavy with MYSQL because this is a simple single-user program, but i'd like to use it from multiple machines.
Thanks.
I don't know if it can be done on MAC, on Debian I have to mount the samba directory with the nobrl option.
From mount.cifs(8):
nobrl
Do not send byte range lock requests to the server. This is
necessary for certain applications that break with cifs
style mandatory byte range locks (and most cifs servers do
not yet support requesting advisory byte range locks).
Read the sqlite FAQ: http://www.sqlite.org/faq.html#q5
"People who have a lot of experience
with Windows tell me that file locking
of network files is very buggy and is
not dependable. If what they say is
true, sharing an SQLite database
between two or more Windows machines
might cause unexpected problems."
So it doesn't work on Windows, it doesn't tell about MAC.
Possibly it fails to lock the file over the network, I think you use SMB protocol so the bugginess comes with the package. If you would like to use SQLite over the network see SQLite Network for alternatives.
I've had a similar problem and I solved it by installing a newer sqlite version. Since Python 2.6 the problem has disappeared too because it uses a newer sqlite dll.
Thank you Carlos. Cherrytree depends on SQLite, and for some reason it recently stopped working with my samba-mounted SQLite database file, complaining about a locked database. Adding "nobrl" to my ubuntu fstab entry solved the problem.
//192.168.3.122/Files /mnt/Files cifs username=public,password=asdf,rw,noperm,nobrl 0 0

How can I uninstall Win32 assemblies and cleanup WinSxS?

After a lot of trial and error (mostly due to lack of documentation and examples) I have managed to create MSI installers that install custom DLLs to WinSxS as side-by-side assembly. There is only one problem: Uninstalling leaves all files (DLLs, manifests and catalogs) in the WinSxS directory. How can or should I best clean that up? I know for sure that nothing else references it.
I have read somewhere that WinSxS has a self-scavenging process that cleans up over time but I could not find more information about that. Can you manually invoke this to clean up stuff?
The only other way I see is manually deleting those bits. First you have to change the owner of all files (assembly, catalog, manifest and their respective directory) from SYSTEM to an administrator account, adjust the permissions and delete them. There are also pieces left in the registry (I think HKLM\COMPONENTS\DerivedData\Components may be one place), but since WinSxS should be treated as opaque it is hard to find any information.
Scavenging isn't exposed anywhere that I know of. I'm not even sure when it is kicked off automatically. Maybe on uninstall of a service pack? Maybe some tool admins can run? I really forget.
Anyway, my suggestion is don't fight it. There are so many twisty turns down there that it just isn't worth trying to get the disk space back. Once uninstalled the bits still in the SxS cache will not be activated so they are just wasting space.
It's a dumb design but blame Microsoft and don't try to overcompensate.
Here is an article, it's kinda complete guide to WinSxS.
So, shortly, you can only uninstall some components (all their versions are in this folder), and you can run Service Pack bridge burning utility (in Vista it is named VSP1CLN.EXE and shipped with SP1). Note, that after execution, you shouldn't be able to uninstall SP or any components to state, prior to SP release date.
No-one is convinced you can - short of a complete reinstall, your bloaty WinSxS directory is there to stay.
There's been a long "discussion" of the problem on technet.
There is no documentation of the format, or any instructions how to remove files that are no longer needed - MS seems to think that disc space is cheap. There is a self-scavenging feature, but no-one's convinced it works, or if it does, it is very conservative (as you'd hope as you don't want it to break your OS)
You can tell is the scavenger is working by checking the "C:\Windows\winsxs\Temp\PendingDeletes." folder, as this is where files are moved by windows update or an installer moves them to - the scavenger just deletes the files in here.
You'll notice that after you uninstall your assembly, while the files are still there, they can no longer be bound to - so they are just "staged", or cached, but not really installed.
Rob & gbjbaanb are correct - you cannot manually invoke a scavenge yourself. Don't try to delete the files yourself - there are multiple places in the registry where they are registered, DerivedData\Components being only one of the many references.
I think the rule for Vista is scavenging is kicked off by the TrustedInstaller service after 10 minutes of machine inactivity, after the last servicing operation (service pack, hotfix, etc). But it's very fickle, so it doesn't run as often as it should. So just be patient, and the files will disappear on their own.
Well i was having some issues as i have an 80GB SSD for my windows and the WinSxs folder was about 12gb's
I was searching the net and i found this command:
DISM.exe /online /Cleanup-Image /spsuperseded
And now my WinSxs is 7gb which was wonderful news.
There are a few updates regarding the cleanup method that apply to newer OS. Check http://www.karafilis.net/winsxs-cleanup

Resources