Seems CurrentProject.ImportExportSpecifications("import-CP_Commandes").Execute has sometimes a strange behaviour.
I need to import daily one or several csv files that will be inserted into an access database. Each file is previously copied in a file named CP_Commandes.csv. I have written a VBA function within Access to do the job.
I have created the necessary importexport specification (named import-CP_Commandes). The relevant xml is stored in a txt file. It works fine with the first file, i.e. the CP_commandes.csv is imported into the relevant Access table (x_temp_commandes). Problem is, after copying the next file into CP_commandes.csv, Executing the import does just nothing, without any error. It looks like Access considers it has already done the job ! Even after stopping the program and re-launching, same result : x_temp_commandes is empty !
After some time, maybe an hour or more, I cant' be more precise, the second file is treated OK ...
I tried different solutions for file #2 to cope with this problem.
Solution 1 :
CurrentProject.ImportExportSpecifications("import-CP_Commandes").delete
DoEvents
CurrentProject.ImportExportSpecifications.add "import-CP_Commandes",ch '(ch previously loaded from the relevant xml file)
DoEvents
CurrentProject.ImportExportSpecifications("import-CP_Commandes").Execute
=> no error, x_temp_commandes empty
Solution 2 :
Same as solution 1, but with ch modified to specify the actual file to be imported instead of CP_Commandes.csv => no error, x_temp_commandes empty.
But sometimes, the program without ANY modification is able to import 2 or 3 files without problem.
Hence my question : Is there something I can do to make sure the work will be done ?
Sorry I finally figured it out - I was not closing the recordset which I oped with
set rs=CurrentDb.OpenRecordset("x_Temp_CP_Commandes")
For this reason, Import did not work - but it did not mention an error. By applying an rs.close after this, the following files import correctly.
Related
I need to write out a file to a certain path that contains a special character in R. the path is something like this: C:/Users/Technology ＆ Innovation/Webscraping files/US_data/data
It works totally fine when I access this path through python, but I cannot access the same path in R. And I cannot change this path name or remove '＆' as this path is used by a lot of people. Does anyone have a good idea on how to solve it?
I found out it is '＆' which has subtle difference from '&' that we usually type in through English Keyboard. May be that's the reason causing the problem?
Here is what I have tried:
write.csv(df, 'C:/Users/Technology ＆ Innovation/Webscraping files/US_data/data/file.csv').
write.csv(df, 'C:\\Users\\Technology ＆ Innovation\\Webscraping files\\US_data/data/file.csv')
Not matter whether I try to read or write a file, it is not working in my case.
I also tried reset the working directory path and got the error message:
Error in setwd("C:/Users/Technology & Innovation/Webscraping files/US_data/data") : cannot change working directory
Write it like this
C:\\Users\\Technology ＆ Innovation\\Webscraping files\\US_data\\data
also, you can change your current directory.
Changing your current directory will help you because you can write read.csv("filename.csv") or write.csv(name_of_file, "filename.csv") as it is without mentioning path.
If you have to write a file you have to use syntax properly.
write.csv(C:\\Users\\Technology ＆ Innovation\\Webscraping files\\US_data\\data,"filename.csv")
I am trying to run an R-script file using windows task scheduler that runs it every two hours. What I am trying to do is gather some tweets through Twitter API and run a sentiment analysis that produces two graphs and saves it in a directory. The problem is, when the script is run again it replaces the already existing files with that name in the directory.
As an example, when I used the pdf("file") function, it ran fine for the first time as no file with that name already existED in the directory. Problem is I want the R-script to be running every other hour. So, I need some solution that creates a new file in the directory instead of replacing that file. Just like what happens when a file is downloaded multiple times from Google Chrome.
I'd just time-stamp the file name.
> filename = paste("output-",now(),sep="")
> filename
[1] "output-2014-08-21 16:02:45"
Use any of the standard date formatting functions to customise to taste - maybe you don't want spaces and colons in your file names:
> filename = paste("output-",format(Sys.time(), "%a-%b-%d-%H-%M-%S-%Y"),sep="")
> filename
[1] "output-Thu-Aug-21-16-03-30-2014"
If you want the behaviour of adding a number to the file name, then something like this:
serialNext = function(prefix){
if(!file.exists(prefix)){return(prefix)}
i=1
repeat {
f = paste(prefix,i,sep=".")
if(!file.exists(f)){return(f)}
i=i+1
}
}
Usage. First, "foo" doesn't exist, so it returns "foo":
> serialNext("foo")
[1] "foo"
Write a file called "foo":
> cat("fnord",file="foo")
Now it returns "foo.1":
> serialNext("foo")
[1] "foo.1"
Create that, then it returns "foo.2" and so on...
> cat("fnord",file="foo.1")
> serialNext("foo")
[1] "foo.2"
This kind of thing can break if more than one process might be writing a new file though - if both processes check at the same time there's a window of opportunity where both processes don't see "foo.2" and think they can both create it. The same thing will happen with timestamps if you have two processes trying to write new files at the same time.
Both these issues can be resolved by generating a random UUID and pasting that on the filename, otherwise you need something that's atomic at the operating system level.
But for a twice-hourly job I reckon a timestamp down to minutes is probably enough.
See ?files for file manipulation functions. You can check if file exists with file.exists, and then either rename the existing file, or create a different name for the new one.
I need to import the Geonames database (http://download.geonames.org/export/dump/) into SQLite (file is about a gigabyte in size, ±8,000,000 records, tab-delimited).
I'm using the built-in SQLite-possibilities of Mac OS X, accessed through terminal. All goes well, until record 381174 (tested with older file, the exact number varies slightly depending on the exact version of the Geonames database, as it is updated every few days), where the error "expected 19 columns of data but found 18" is displayed.
The exact line causing the problem is:
126704 Gora Kyumyurkey Gora Kyumyurkey Gora Kemyurkey,Gora
Kyamyar-Kup,Gora Kyumyurkey,Gora Këmyurkëy,Komur Qu",Komur
Qu',Komurkoy Dagi,Komūr Qū’,Komūr Qū”,Kummer Kid,Kömürköy Dağı,kumwr
qwʾ,كُمور
قوء 38.73335 48.24133 T MT AZ AZ 00 0 2471 Asia/Baku 2014-03-05
I've tested various countries separately, and the western countries all completely imported without a problem, causing me to believe the problem is somewhere in the exotic characters used in some entries. (I've put this line into a separate file and tested with several other database-programs, some did give an error, some imported without a problem).
How do I solve this error, or are there other ways to import the file?
Thanks for your help and let me know if you need more information.
Regarding the question title, a preliminary search resulted in
the GeoNames format description ("tab-delimited text in utf8 encoding")
https://download.geonames.org/export/dump/readme.txt
some libraries (untested):
Perl: https://github.com/mjradwin/geonames-sqlite (+ autocomplete demo JavaScript/PHP)
PHP: https://github.com/robotamer/geonames-to-sqlite
Python: https://github.com/commodo/geonames-dump-to-sqlite
GUI (mentioned by #charlest):
https://github.com/sqlitebrowser/sqlitebrowser/
The SQLite tools have import capability as well:
https://sqlite.org/cli.html#csv_import
It looks like a bi-directional text issue. "كُمور قوء" is expected to be at the end of the comma-separated alternate name list. However, on account of it being dextrosinistral (or RTL), it's displaying on the wrong side of the latitude and longitude values.
I don't have visibility of your import method, but it seems likely to me that that's why it thinks a column is missing.
I found the same problem using the script from the geonames forum here: http://forum.geonames.org/gforum/posts/list/32139.page
Despite adjusting the script to run on Mac OS X (Sierra 10.12.6) I was getting the same errors. But thanks to the script author since it helped me get the sqlite database file created.
After a little while I decided to use the sqlite DB Browser for SQLite (version 3.11.2) rather than continue with the script.
I had errors with this method as well and found that I had to set the "Quote character" setting in the import dialog to the blank state. Once that was done the import from the FULL allCountries.txt file ran to completion taking just under an hour on my MacBookPro (an old one but with SSD).
Although I have not dived in deeper I am assuming that the geonames text files must not be quote parsed in any way. Each line simply needs to be handled as tab delimited UTF-8 strings.
At the time of writing allCountries.txt is 1.5GB with 11,930,517 records. SQLite database file is just short of 3GB.
Hope that helps.
UPDATE 1:
Further investigation has revealed that it is indeed due to the embedded quotes in the geonames files, and looking here: https://sqlite.org/quirks.html#dblquote shows that SQLite has problems with quotes. Hence you need to be able to switch off quote parsing in SQLite.
Despite the 3.11.2 version of DB Browser being based on SQLite 3.27.2 which does not have the required mods to ignore the quotes, I can only assume it must be escaping the quotes when you set the "Quote character" to blank.
I'm writing a program that performs several tests on a hardware unit, and logs both the results of each test and the steps taken to perform the test. The trick is that I want the program to log these results to a text file as they become available, so that if the program crashes the results that had been obtained are not lost, and the log can help debug the crash.
For example, assume a program consisting of two tests. If the program has finished the first test and is working on the second, the log file would look like:
Results:
Test 1 Result A: Passed
Test 1 Result B: 1.5 Volts
Log:
Setting up instruments.
Beginning test 1.
[Steps in test 1]
Finished test 1.
Beginning test 2.
[whatever test 2 steps have been completed]
Once the second test has finished, the log file would look like this:
Results:
Test 1 Result A: Passed
Test 1 Result B: 1.5 Volts
Test 2 Result A: Passed
Test 2 Result B: 2.0 Volts
Log:
Setting up instruments.
Beginning test 1.
[Steps in test 1]
Finished test 1.
Beginning test 2.
[Steps in test 2]
Finished test 2.
All tests complete.
How would I go about doing this? I've been looking at the help files for QFile and QTextStream, but I'm not seeing a way to insert text in the middle of existing text. I don't want to create separate files and merge them at the end because I'd end up with separate files in the event of a crash. I also don't want to write the file from scratch every time a change is made because it seems like there should be a faster, more elegant way of doing this.
QFile.readAll will read the entire file into a QByteArray.
On the QByteArray you can then use insert to insert text in the middle,
and then write it back to file again.
Or you could use the classic c style that can modify files in the middle with the help of filepointers.
As #Roku pointed out, there is no built in way to insert data in a file with a rewrite. However if you know the size of the region, i.e., if the text you want to write has a fixed length, then you can write an empty space in the file and replace it later. Check
this discussion in overwriting part of a file.
I ended up going with the "write the file from scratch" method that I mentioned being hesitant about in my question. The benefit of this technique is that it results in a single file, even in the event of a crash since the log and the results are never placed in different files to begin with. Additionally, rewriting the file only happens when adding new results (an infrequent occurrence), whereas updating the log means simply appending text to the file as usual. I'm still a bit surprised that there isn't a way to have the OS insert text into a file for you.
Oh, and for those of you who absolutely must have this functionality as efficiently as possible, the following might be of use:
http://www.codeproject.com/Articles/17716/Insert-Text-into-Existing-Files-in-C-Without-Temp
You just cannot add more stuff in the middle of a file. I would go with two separate files, another for the results and another for the logs.
I can do that file-by-file with Save As Encoding in Visual Studio, but I'd like to make this in one click. Is it possible?
I know, some will start bashing on me:
download a smalltalk IDE (such as ST/X),
open a workspace,
type in:
'yourDirectoryHere' asFilename directoryContentsAsFilenamesDo:[:oldFileName |
|cyrString utfString newFile|
cyrString := oldFileName contentsAsString.
utfString := CharacterEncoder encodeString:cyrString from:#'iso8859-5' into:#'utf'.
newFile := oldFile withSuffix:'utf'.
newFile contents:utfString.
].
that will convert all files in the given directory and create corresponding .utf files without affecting the original files. Even if you normally do not use smalltalk, for this type of actions, smalltalk is a perfect scripting environment.
I know, most of you don't read smalltalk, but the code should be readable even for non-smalltalkers and a corresponding perl/python/java/c# piece of code also written and executed in 1 minute or so, taking the above as a guide. I guess all current languages provide something similar to the CharacterEncoder above.