I am in a unix env and I need to append below content to a txt file without starting at a new line:
the content is:
set value = 9.0.1.3 || set var = 12.4
the txt file content is:
echo on
set name = 'charlie' ||
but when I use the command 'cat', I can append the content to the txt file, but the content will start from a new line, like this:
echo on
set name = 'charlie' ||
set value = 9.0.1.3 || set var = 12.4
well, my goal is to get it like this:
echo on
set name = 'charlie' || set value = 9.0.1.3 || set var = 12.4
What command should I use to reach my goal? thx.
sed -i '$s/$/ set value = 9.0.1.3 || set var = 12.4/' FILE
That is inplace editing FILE (by -i, if you need a backup, use -i.BACKUPEXTENSION). The 1st $ after the ' addresses the last line, then ssubstitute the empty $tring on it at the end of the line with your string.
If it should be done on other line(s) than the last, you can address the substitute command with line numbers, regex, etc.
By default cat is not putting any new line charecter in the end of the file. If you already have a newline character in the end of the file then that will be concatinated.
>echo "TTTT\c">test1
>echo "GGGG\c">test2
>cat test1 test2
TTTTGGGG>
Please remove the newline character at the end of your file and then run the cat command.
You can remove the newline character at the end of file using below command:
awk 'BEGIN{flag=0}{if(flag==1)print a;flag=1;a=$0}END{printf("%s",a)}' FILE_NAME >FILE_NAME
Related
I have a file named param1.txt which contains certain variables. I have another file as source1.txt which contains place holders. I want to replace the place holders with the values of the variables that I get from the parameter file.
I have basically hard coded the script where the variable names in the parameter.txt file is known before hand. I want to know a dynamic solution to the problem where the variable names will not be known beforehand. In other words, is there any way to find out the variable names in a file using the source command in UNIX?
Here is my script and the files.
Script:
#!/bin/bash
source /root/parameters/param1.txt
sed "s/{DB_NAME}/$DB_NAME/gI;
s/{PLANT_NAME}/$PLANT_NAME/gI" \
/root/sources/source1.txt >
/root/parameters/Output.txt`
param1.txt:
PLANT_NAME=abc
DB_NAME=gef
source1.txt:
kdashkdhkasdkj {PLANT_NAME}
jhdbjhasdjdhas kashdkahdk asdkhakdshk
hfkahfkajdfk ljsadjalsdj {PLANT_NAME}
{DB_NAME}
I cannot comment since I don't have enough points.
But is it correct that this is what you're looking for:
How to reference a file for variables using Bash?
Your problem statement isn't very clear to me. Perhaps you can simplify your problem and desired state.
Don't understand why you try to source param1.txt.
You can try with this awk :
awk '
NR == FNR {
a[$1] = $2
next
}
{
for ( i = 1 ; i <= NF ; i++ ) {
b = $i
gsub ( "^{|}$" , "" , b )
if ( b in a )
sub ( "{" b "}" , a[b] , $i )
}
} 1' FS='=' param1.txt FS=" " source1.txt
We know U-SQL supports directory and filename pattern matching while extracting the files. What I wanted to know does it support pattern matching based on date of creation of the file in ADLS (without implementing custom extractors).
Say a folder contains files created across months (filenames don't have date as part of the filename), is there a way to pull only files of a particular month.
The U-SQL EXTRACT operator is not aware of any metadata (such as create date) about a file - only the filename.
You could probably build a solution using the .NET SDK. For something rather simple you could use PowerShell to create a file which will contain all the files that meet your date time criteria. Then consume the content as desired.
# Log in to your Azure account
Login-AzureRmAccount
# Modify variables as required
$DataLakeStoreAccount = "<yourDataLakeStoreAccountNameHere>";
$DataLakeAnalyticsAccount = <yourDataLakeAnalyticsAccountNameHere>";
$DataLakeStorePath = "/Samples/Data/AmbulanceData/"; #modify as desired
$outputFile = "Samples/Outputs/ReferenceGuide/filteredFiles.csv"; #modify as desired
$filterDate = "2016-11-22";
$jobName = "GetFiles";
# Query directory and build main body of script. Note, there is a csv filter.
[string]$body =
"#initial =
SELECT * FROM
(VALUES
" +
(Get-AzureRmDataLakeStoreChildItem -Account $DataLakeStoreAccount -Path $DataLakeStorePath |
Where {$_.Name -like "*.csv" -and $_.Type -eq "FILE"} | foreach {
"(""" + $DataLakeStorePath + $_.Name + """, (DateTime)FILE.CREATED(""" + $DataLakeStorePath + $_.Name + """)), `r`n" });
# formattig, add column names
$body =
$body.Substring(0,$body.Length-4) + "
) AS T(fileName, createDate);";
# U-SQL query and OUTPUT statement
[string]$output =
"
// filter results based on desired time frame
#filtered =
SELECT fileName
FROM #initial
WHERE createDate.ToString(""yyyy-MM-dd"") == ""$filterDate"";
OUTPUT #filtered
TO ""$outputFile""
USING Outputters.Csv();";
# bring it all together
$script = $body + $output;
#Execute job
$jobInfo = Submit-AzureRmDataLakeAnalyticsJob -Account $DataLakeAnalyticsAccount -Name $jobName -Script $script -DegreeOfParallelism 1
#check job progress
Get-AzureRmDataLakeAnalyticsJob -Account $DataLakeAnalyticsAccount -JobId $jobInfo.JobId -ErrorAction SilentlyContinue;
Write-Host "You now have a list of desired files to check # " $outputFile
Currently there is no way to access or use file meta data properties. Please add your vote and use case to the following feedback item: https://feedback.azure.com/forums/327234-data-lake/suggestions/10948392-support-functionality-to-handle-file-properties-fr
it's been a while since this question was asked, and I'm not sure if this is what you were looking for originally, but now you can use the FILE.MODIFIED U-SQL function:
DECLARE #watermark string = "2018-08-16T18:12:03";
SET ##FeaturePreviews="InputFileGrouping:on";
DECLARE #file_set_path string = "adl://adls.azuredatalakestore.net/stage/InputSample.tsv";
#input =
EXTRACT [columnA] int?,
[columnB] string
FROM #file_set_path
USING Extractors.Tsv(skipFirstNRows : 1, silent : true);
#result =
SELECT *, FILE.MODIFIED(#file_set_path) AS FileModifiedDate
FROM #input
WHERE FILE.MODIFIED(#file_set_path) > DateTime.ParseExact(#watermark, "yyyy-MM-ddTHH:mm:ss", NULL);
OUTPUT #result TO "adl://ADLS.azuredatalakestore.net/stage/OutputSample.tsv" USING Outputters.Tsv(outputHeader:true);
The U-SQL built-in function is documented here:
https://msdn.microsoft.com/en-us/azure/data-lake-analytics/u-sql/file-modified-u-sql
I am trying to load a bunch of Java files into a staging table using SQL Loader. I keep getting the subject line error and I'm not sure why.
My executable file that I am trying to run looks like this.
for i in `find <files in certain directory.java>`
do
echo "File name = ${i}"
COMMAND='sed'; ARG='s/XXXX/${i}/'
echo $COMMAND '; ' $ARG
cat test_load.ctl | $COMMAND "$ARG" > test_load_2.ctl
sqlldr <user>/<password> control=test_load_2.ctl log=<file_name>.log
done
My test_load.ctl file looks like this:
I'm trying to replace the XXXX INFILE with the looped through files in the Java directory above.
LOAD DATA
INFILE 'XXXX'
BADFILE '/<directory>/<filename>.bad'
DISCARDFILE '/<directory>/<filename>.dsc'
APPEND INTO TABLE "<SCHEMA>"."<TABLE_NAME>"
TRAILING NULLCOLS
(
id sequence (1, 1),
raw_string position (1:4000) char(4000),
file_name,
dir_path,
load_date sysdate,
line_number sequence (1, 1)
)
My test_load_2.ctl file looks like this:
LOAD DATA
INFILE '${i}'
BADFILE '/<directory>/<filename>.bad'
DISCARDFILE '/<directory>/<filename>.dsc'
APPEND INTO TABLE "<SCHEMA>"."<TABLE_NAME>"
TRAILING NULLCOLS
(
id sequence (1, 1),
raw_string position (1:4000) char(4000),
file_name,
dir_path,
load_date sysdate,
line_number sequence (1, 1)
)
I keep getting this error:
SQL*Loader-503: Error appending extension to file ()
Additional information: 7217
I'm pretty sure there is an issue with the INFILE parameter in the test_load_2.ctl file, but I am not 100% certain how to fix this?
Also I'm probably doing something wrong in the executable file as well.
Any suggestions?
Thank you!
No variable expansion is going to happen in the control file. Specify the INFILE on the commandline instead using the DATA argument.
https://docs.oracle.com/cd/E11882_01/server.112/e22490/ldr_params.htm#SUTIL004
I want to use the QString.split(' ') method to split a input command into the command
QStringList commandList = command.split(' ');
however command has a UNIX path at the end of it. i.e. it looks something like
QString command = new QString("caommand -a -b /path/to\ specific/file");
the path command is specified by the user at runtime(the user escapes any spaces in the path). For some reason command.split(' '); does not escape the spaces.
I am new to QT, how does it escape spaces?
Thanks for any help
You can use QDir::toNativeSeparators() to convert it to unix style. And split received result by spaces, though you have got to figure out where are the spaces between commands and where are the possible spaces in filename
For example:
QString myUnixPath = QDir::toNativeSeparators("/home/path with spaces/");
will return unix style path, while
QString qtPath = QDir::fromNativeSeparators("/path/with\ spaces/");
will return /path with spaces/
I'm very new to VBS and experimenting with removing the read only attribute from a directory recursively.
It's removing the read only attribute for files but not for the directories. Also The files within these directories seem to have lost their associated program link and are now all showing as an unregistered file type. Any help is greatly appreciated.
Update: I can see why the files have lost their associations now. It's because the . which separates the name from the extension has been removed! Doh! Ideally I'd like to rename the filename only.
re.Pattern = "[_.]"
re.IgnoreCase = True
re.Global = True
RemoveReadonlyRecursive("T:\Torrents\")
Sub RemoveReadonlyRecursive(DirPath)
ReadOnly = 1
Set oFld = FSO.GetFolder(DirPath)
For Each oFile in oFld.Files
If oFile.Attributes AND ReadOnly Then
oFile.Attributes = oFile.Attributes XOR ReadOnly
End If
If re.Test(oFile.Name) Then
oFile.Name = re.Replace(oFile.Name, " ")
End If
Next
For Each oSubFld in oFld.SubFolders
If oSubFld.Attributes AND ReadOnly Then
oSubFld.Attributes = oSubFld.Attributes XOR ReadOnly
End If
If re.Test(oSubFld.Name) Then
oSubFld.Name = re.Replace(oSubFld.Name, " ")
End If
RemoveReadonlyRecursive(oSubFld.Path)
Next
End Sub
It seems you want to automate a repeatable action by a script. Why don't you use the attrib command to do that for you:
attrib -r "T:\Torrents\*.*" /S
You can place that in a batch file if you want to attach it to an clickable icon.
EDIT:
To run it from VBScript silently:
Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.Run "attrib -r ""T:\Torrents\*.*"" /S", 0, true)
EDIT2:
To replace everything except the last period, use a regular expression like:
filename = "my file.name.001.2012.extension"
Set regEx = New RegExp
' Make two captures:
' 1. Everything except the last dot
' 2. The last dot and after that everything that is not a dot
regEx.Pattern = "^(.*)(\.[^.]+)$" ' Make two captures:
' Replace everything that is a dot in the first capture with nothing and append the second capture
For each match in regEx.Execute(filename)
newFileName = replace(match.submatches(0), ".", "") & match.submatches(1)
Next