msys / mingw command line requires or rejects exe extension based on whether stdout is redirected with ConEmu console - console

The underlying cause of this problem is described elsewhere, with partial workarounds provided.
For example: stdout is not a tty and stdin is not a tty
An example of a command line I'm having problems with in MSYS2 or MINGW64 environments is this:
# psql -c '\d' | grep public
stdout is not a tty
Here's the output, if issued without piping to grep:
# psql -c '\d'
List of relations
Schema | Name | Type | Owner
--------+----------------+----------+----------
public | tagname | table | postgres
public | tagname_id_seq | sequence | postgres
(2 rows)
In order to redirect stdout, it's apparently necessary to edit the command line, changing psql to psql.exe. This modification does the trick:
# psql.exe -c '\d' | grep public
public | tagname | table | postgres
public | tagname_id_seq | sequence | postgres
This version works, whether or not stdout is redirected:
# psql.exe -c '\d'
List of relations
Schema | Name | Type | Owner
--------+----------------+----------+----------
public | tagname | table | postgres
public | tagname_id_seq | sequence | postgres
(2 rows)
Note that the problem only exists for some programs. For example, /usr/bin/find is indifferent to whether the .exe extension is specified. Also, the cygwin version of psql.exe does not suffer from this limitation.
The workaround of appending .exe could be hidden with an alias if you could always call psql as psql.exe, but there are problems with interactive sessions when called with the extension. (in ConEmu terminal under MSYS_NT-10.0-19042, for example)
So my question is this: is it possible to create a wrapper program (for example, in golang or C, or a shell script) to hide this problem? The goal would be to support optionally redirectable command lines without requiring the .exe extension.
Assuming the wrapper is named "fixtty", it would spawn a command line that could be redirected or not. In other words, neither of these command lines would fail with an error message:
# fixtty psql -c '\d'
# fixtty psql -c '\d' | grep public

A wrapper script (call it fixtty) that hides the problem:
#!/bin/bash
if [ -t 1 ]; then
# stdout is a terminal
"$#"
else
# stdout is a file or pipe
PROG=$1 ; shift
set -- "$PROG.exe" "$#" # append .exe suffix
"$#"
fi
Usage:
$ fixtty psql -c '\d' # unfiltered `stdout`
$ fixtty psql -c '\d' | grep public # filtered
$ fixtty psql # launch interactive session

Related

How to print unix tool version in BusyBox container?

I can't figure out how to print (unix tool) versions within a BusyBox container:
$ docker run -it quay.io/quay/busybox:latest
$ awk --version
awk: unrecognized option `--version'
BusyBox v1.32.0 (2020-08-31 17:40:13 UTC) multi-call binary.
Usage: awk [OPTIONS] [AWK_PROGRAM] [FILE]...
-v VAR=VAL Set variable
-F SEP Use SEP as field separator
-f FILE Read program from FILE
-e AWK_PROGRAM
$ cut --version
cut: unrecognized option `--version'
BusyBox v1.32.0 (2020-08-31 17:40:13 UTC) multi-call binary.
Usage: cut [OPTIONS] [FILE]...
Print selected fields from each input FILE to stdout
-b LIST Output only bytes from LIST
-c LIST Output only characters from LIST
-d CHAR Use CHAR instead of tab as the field delimiter
-s Output only the lines containing delimiter
-f N Print only these fields
-n Ignored
Any suggestions? Many mulled containers are built on top of BusyBox, best I get on top of this.
Thanks
busybox is a single program which acts as one of various tools depending on what name was used to call it. As you can see in the question, it shows its version as BusyBox v1.32.0.
Check which tools are (symbolic) links to busybox. All these are the same program and therefore have the same version, so you might only need the version of busybox and a list of commands linked to it.
According to https://unix.stackexchange.com/q/15895/330217 the best way to display the version of busybox is
busybox | head -1

jq -c fails in AWS pipeline

I use cat file.json | jq -c to print out a minified json file in the logs in an AWS pipeline step. I've tested that it works locally but in the pipeline it fails and prints out the system usage:
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]
...
Why does it fail in the pipeline but not locally?
That's not the right question. jq -c (with no further argument) should fail. According to the very output you obtained from jq, jq always requires at least one argument: a program.[1] That fact that it sometimes doesn't fail when no program is provided is a bug. One that appears to have been fixed in 1.6.[2]
If you simply want to reformat the JSON, you can use the trivial program ..
cat input.json | jq -c .
If the input really is a file, the following is better:
jq -c . input.json
Or the name of a file containing the program if -f is used.
Meaning I wasn't able to reproduce the lack of error in 1.6 after trying using Windows, Cygwin and Ubuntu builds.
The solution was to use cat file.json | jq -c . (note the dot that's added) in AWS.
After some experimenting, I found out that this happens only in jq 1.5 (I had jq 1.6 locally), and only when using jq -c instead of jq -c . in a detached terminal. My AWS pipeline uses jq 1.5.
I reproduced it locally by downloading jq 1.5 and running:
$ cat input.json | ./jq1.5 -c >output.txt &
[1] 822
jq - commandline JSON processor [version 1.5]
Usage: C:\Workspace\jq\jq1.5.exe [options] <jq filter> [file...]
Note that the ampersand executes it in the background which simulates a detached terminal.
What works:
cat input.json | ./jq1.6 -c >output.txt &
cat input.json | ./jq1.5 -c . >output.txt &
cat input.json | ./jq1.5 -c >output.txt
To sum up, if using jq1.5 in a detached terminal, also use the dot.

Argument in Dockerfile not passed as executed command

In my Dockerfile I'm trying to download the latest WordPress version without any content inside it, but I'm having trouble automating the latest version number so that I don't have to manually change it when the new version of WordPress comes out.
In my Dockerfile I have
ARG LATESTWPVER="$(curl -s https://api.wordpress.org/core/version-check/1.5/ | head -n 4 | tail -n 1)"
ADD $(https://downloads.wordpress.org/release/wordpress-$LATESTWPVER-no-content.zip) /var/www/latest.zip
But the problem is that my LATESTWPVER is not 4.9.8, and I get the error
ADD failed: stat /var/lib/docker/tmp/docker-builder962069305/$(https:/downloads.wordpress.org/release/wordpress-$(curl -s https:/api.wordpress.org/core/version-check/1.5/ | head -n 4 | tail -n 1)-no-content.zip): no such file or directory
It passes the entire command and I'd like to have the output of that command.
In my shell file the
#!/bin/bash
WP_LATEST="$(curl -s https://api.wordpress.org/core/version-check/1.5/ | head -n 4 | tail -n 1)"
echo $WP_LATEST
will return the number 4.9.8.
From the error, I'm guessing that you can only assign something to the variable, but not execute it. Is there a way to execute a command and assign it to a variable and pass it as an argument?
A Dockerfile is not a shell or a build script, so it will not execute what you pass in ARG. There is a workaround - define the version as an ARG and pass that during build.
Dockerfile:
--
FROM ubuntu:latest
ARG LATESTWPVER
RUN echo $LATESTWPVER
ADD https://downloads.wordpress.org/release/wordpress-$LATESTWPVER-no-content.zip /var/www/latest.zip
docker build --build-arg LATESTWPVER=`curl -s https://api.wordpress.org/core/version-check/1.5/ | head -n 4 | tail -n 1` .
Sending build context to Docker daemon 6.656kB
Step 1/4 : FROM ubuntu:latest
---> 113a43faa138
Step 2/4 : ARG LATESTWPVER
---> Using cache
---> 64f47dcfe7fa
Step 3/4 : RUN echo $LATESTWPVER
---> Running in eb5fdd005d77
4.9.8
Removing intermediate container eb5fdd005d77
---> 1015629b927e
Step 4/4 : ADD https://downloads.wordpress.org/release/wordpress-$LATESTWPVER-no-content.zip /var/www/latest.zip
Downloading [==================================================>] 7.118MB/7.118MB
---> 72f0d3790e51
Successfully built 72f0d3790e51

grep a log file for specific word that occured on today

I have to search how many times we have received any specific exceptions on current date. I am using below command but it doesn't work.
This command shows ClassCastException or NumberFormatException that occurred until now.
I just wanted to know how many times ClassCastException or NumberFormatException occurred in today's date only.
grep $(date +"%Y-%m-%d") /*.* |grep -ioh "ClasscastException\|NumberFormatException" /logs/*.* | sort | uniq -c | sort -r
grep -ioh "ClasscastException\|NumberFormatException" /logs/*.* | sort | uniq -c | sort -r
Above command gave me no of count for ClassCastException and NumberFormatException in log file for all dates. I just want for today's date count.
The first command should work after removing the /logs/*.* part
grep $(date +"%Y-%m-%d") /∗.∗ |grep -ioh "ClasscastException\|NumberFormatException" /logs/∗.∗ | sort | uniq -c | sort -r
grep works on the files that are given as argument, otherwise, it defaults to stdin.
Since files are supplied as argument to the the 2nd grep in this pipeline, it is discarding the output from the 1st grep and looking for the pattern in all files under log directory again.

single quotes not working in shell script

I have a .bash_profile script and I can't get the following to work
alias lsls='ls -l | sort -n +4'
when I type the alias lsls
it does the sort but then posts this error message
"-bash: +4: command not found"
How do I get the alias to work with '+4'?
It works when type ls -l | sort -n +4 in the command line
I'm in OS X 10.4
Thanks for any help
bash-4.0$ ls -l | sort -n +4
sort: open failed: +4: No such file or directory
You need ls -l | sort -n -k 5, gnu sort is different from bsd sort
alias lsls='ls -l | sort -n -k 5'
Edit: updated to reflect change from 0 based indexing to 1 based indexing, thanks Matthew.
alias lsls='ls -l | sort -n +4' should work fine with the sort in OS X 10.4 (which does support that syntax).
when I type the alias lsls it does the sort but then posts this error message "-bash: +4: command not found"
Is it possible that you inserted a stray newline when editing your .bash_profile? e.g. if you ended up with something like this:
alias lsls='ls -l | sort -n
+4'
...that might explain the error message.
As an aside, you can get the same effect without piping through sort at all, using:
ls -lrS
This link discusses a very similar alias containing a pipe.
The problem may not have been the pipe, but the interesting solution was to use a function.

Resources