Is it possible to save a mapping file which SSIS can use to decide the data type based on the column names rather than going and updating it in the Advanced section of the 'Flat File Connection Manager Editor'. Thank you
This is a common problem faced by every SSIS developer. Whenever you make changes in a flat file connection, you lose all data type mappings and you have to manually edit this by using the advanced editor.
But you can save your life using the following:
Practice #1
When you work with an existing connection, make sure you have the flat file at the reference location of the flat file connection with the same name. If you forget to save it or don't find it, try the second practice.
Practice #2
Follow the steps below before using the SSIS package:
Open package in XML file format.
Find the flat file connection.
Read the file name and path of flat file connection.
Get the output copy of the final output file (usually you can find where SSIS is exporting final output file).
Copy the final output file and rename with the file connection's file name and paste it to the flat file connection location.
Remove all the data from file except the column list (make sure you keep the file format as it is, e.g., CSV or Excel).
Close the XML of the SSIS package & close the package.
Reopen the SSIS package and you saved your life.
This trick works for me in all my cases.
Related
I have two files that come in daily to a shared drive. When they are posted, they come in with the current date as part of the file name. example ( dataset1_12517.txt and dataset2_12517.txt) the next day it posts it will be (dataset1_12617.txt and so on). They are pipe delimited files if that matters.
I am trying to automate a daily merge of these two files to a single excel file that will be overwritten with each merge (file name remains the same) so my tableau dashboard can read the output without having to make a new connection daily. The tricky part is the file names will change daily, but they follow a specific naming convention.
I have access to R Studio. I have not started writing code yet so looking for a place to start or a better solution.
On a Window machine, use the copy or xcopy command lines. There are several variations on how to do it. The jist of it though is that if you supply the right switches, the source file will append to the destination file.
I like using xcopy for this. Supply the destination file name and then a list of source files.
This becomes a batch file and you can run it as a scheduled task or on demand.
This is roughly what it would look it. You may need to check the online docs to choose the right parameter switches.
xcopy C:\SRC\souce_2010235.txt newfile.txt /s
As you play with it, you may even try using a wildcard approach.
xcopy C:\SRC\*.txt newfile.txt /s
See Getting XCOPY to concatenate (append) for more details.
what is the purpose of the R temporary file that is created in every directory where a workspace is saved? What data does it contain and is it safe to delete?
That file is a holding file for save.image() while R waits for its file argument to succeed. From help(save.image) and its safe argument -
safe - logical. If TRUE, a temporary file is used for creating the saved workspace. The temporary file is renamed to file if the save succeeds. This preserves an existing workspace file if the save fails, but at the cost of using extra disk space during the save.
So the file contains the entire workspace image and it is probably best to just leave it there in case R fails to save the workspace normally.
I'm also guessing that if you see this file, R has already failed to rename the file so you may want to search for file and check its contents before deleting the temporary file.
I'm building an application that takes as an input data stored in an Excel sheet. I want the user to be able to select the file they want to load data from, have the application connect to and read from the Excel file stored on the client's machine, and close the connection. Can I do this without uploading the Excel file to the server? I'm able to do everything except for selecting the file using a filedialog box and passing the path & file name to the procedure that connects to the Excel file and processes. I've tried using the file input control but I'm unable to pass the path & file name to the connection string. Any suggestions as to what other routes I might take?
EDIT:
The application essentially takes user input, either via single inputs into textboxes on the page or a bulk upload via an Excel spreadsheet, processes the inputs and spits out a report in Excel format. The only thing displayed on the page are the loaded inputs (via the 2 methods just described) in a listbox that the user can either add or remove additional items.
The short answer is "Only with an ActiveX control in IE unless you write your own plugin for another browser." My opinion is, "you shouldn't."
There is a good discussion already on Stack Overflow: How to read an excel file contents on client side?
The long answer, given the rest of the information you have provided, is that I would recommend the following:
1) Upload the spreadsheet to the server.
2) Extract the data on the server.
3) Return the data to the client in whatever form suits your situation.
4) Clean the original file on the server.
Some further recommendations for you, since you provided so much detail in your comments:
Rather than using a GUID to generate your server-side filename, use a timestamp in the format YYYY-MM-DD-HH-MM-SS-Ticks followed by the original filename.
Instead of deleting the data files immediately, each time you add a file, remove any files older than N days.
This way if you have any issues processing your files in the future, you'll be able to retrieve the uploaded file to your development environment and debug it there. Assuming the data isn't personal information or sensitive in some other way, of course.
Cheers!
I have been implementing an entity in Symfony 2.2 in order to upload files to my server. I followed successfully the steps listed in
http://symfony.com/doc/current/cookbook/doctrine/file_uploads.html
However I need to implement an additional feature, which consists in saving the file along the entity, but not the original one, but the zipped & encrypted one, same as if I'd done that using the command line of linux and then uploaded the generated zip file. I mean, when I'm required to select in my form the file, I choose it as normal, but in the server it'd be stored a zip which contains that file instead of the file itself, and of course when downloading I want the zip as well, so the name in the table has to be the one of the zip file.
I guess it could be accomplished using system calls, allowing PHP to execute a zip command over the file, but I cannot figure it out how exactly. Any help?
Suppose a log file is being written to disk with one extra line appended to it every so often (by a process I have no control over).
I would like to know a clean way to have an R program "watch" the log file, and process a new line when it is written to the log file.
Any advice would by much appreciated.
You can use file.info to get the modification date of a file, just check every so often and take action is the modification date changes. Keeping track of how many lines have already been read will enable you to use scan or read.table to read only the new lines.
You could also delete or move the log file after it is read by your program. The external program will then create a new log file, I assume. Using file.exists you can check if the file has been recreated, and read it when needed. You then add the new data to the already existing data.
I would move the log file to an archive subfolder and read the logfiles as they are created.