OpenStack Swift cannot use bulk operation to auto extract tar file - http

I want to upload many files with a single operation in OpenStack Swift. I find the middleware -- Bulk Operations which can auto extract files from tar compressed file. However, I failed to extract the files from the tar.
I PUT the tar file use the bulk operation like this:
curl -X PUT http://127.0.0.1:8080/v1/AUTH_test/ContainerName/$?extract-archive=tar \
-T theTarName.tar \
-H "Content-Type: text/plain" \
-H "X-Auth-Token: token"
I am sure that the storageURL, tar file path, and token is accurate. But, I didn't get any responses(successes or errors). When I show the objects in the container, I find just one object named 0extract-archive=tar was uploaded, but the files in the tar were not extracted.
I want to know how to extract the tar automatically in OpenStack Swift and all of the files in the tar can be displayed in the container.
Thanks in advance.

The issue is the $? part. $? refers to the exit code of the last command in bash (http://tldp.org/LDP/abs/html/exit-status.html), which I suspect you're using.
If you'd like to use $ as the archive prefix, consider escaping it with \:
$ curl -X PUT \
"http://127.0.0.1:8080/v1/AUTH_test/container/\$?extract-archive=tar" \
-T test.tar \
-H "X-Auth-Token: <token>"
You should get the following output:
Number Files Created: 3
Response Body:
Response Status: 201 Created
Errors:

Related

Create multiple artifactory repositories from json

I want to automate the process of importing existing repositories structure from another Artifactory through .json file.
So far, I have managed to make single repo from json with the following command.
curl -X PUT --insecure -u admin -H "Content-type: application/json" -T repository-config.json "https://artifactory.test.net/artifactory/api/repositories/acqbo-docker-release-local"
Is there a way to import multiple/array of repositories from a single json file and a single curl?
Ended up writing my own bash script for this purpose.
you will have to make a file with the repositories you want to copy:
#!/bin/bash
#############
# This script copies the repository structure from one Artifactory server to another
# repos.list file is required to have repositories that we want to copy, each in new line.
# user with the admin rights is necessary
#############
#Where to copy repos from and to
ARTIFACTORY_SOURCE="https://source.group.net/artifactory/api/repositories/"
ARTIFACTORY_DESTINATION="https://destination.group.net/artifactory/api/repositories/"
NOLINES=$(wc -l < repos.list) #get total nuber of lines in repos.line
COUNTER=1 #Set the counter to 1
while [ $COUNTER -le $NOLINES ] #loops times number of lines in the repos.line
do
REPONAME=$(awk "NR==$COUNTER" repos.list) #get only repo name, line by line
curl -GET --insecure -u admin:adminpass "$ARTIFACTORY_SOURCE$REPONAME" > xrep.json #Obtain data from Artifactory source, repo by repo, and writes it to the xrep.json
curl -X PUT --insecure -u admin:adminpass -H "Content-type: application/json" -T xrep.json "$ARTIFACTORY_DESTINATION$REPONAME" #Sends data from json to the destination Artifactory server
#print in blue color
printf "\e[1;34m$COUNTER repo done\n\e[0m"
((COUNTER++))
done
printf "\e[1;34mAll repos exported!\n\e[0m"

How can I set the version of a raw file in a (Sonatype) Nexus raw repository?

I'm making an automatic script to upload a raw file in Nexus, and I need to set up the version of this file. Is this possible? I've been checking the API but it doesn't seem posible.
The command I'm currently using to upload is:
curl --proxy $my-proxy -v --user 'user:pass' --upload-file ./myRawFile 'http://12.34.56.78:1234/repository/MyRawRepo/LoL/TheUploadedFile'
This command is being used from an automatic script (and working) to upload the file, but I don't know how to set the version.
curl -k -u "xxx:xxx" -H 'Content-Type: multipart/form-data' --data-binary "#output.zip" -X PUT https://nexus.xxx.com/repository/{raw-reponame}/xxx/{version}/output.zip
Version number can be change {version}

where to store request.json file

I am trying out Google video intelligent api by following the quick start guide here. I am using cloud shell to run all the codes. In the guide, to make an annotate video request, I need to create a request.json file.
Where to put this file once it's created?
Thanks very much for your help!
The request.json file in this case should be saved in the same directory that you'll make the curl request from in step 2 of that guide. The curl command points to that file at the end:
curl -s -H 'Content-Type: application/json' \
-H 'Authorization: Bearer ACCESS_TOKEN' \
'https://videointelligence.googleapis.com/v1beta1/videos:annotate' \
-d #request.json
The #request.json means it's looking in the current directory for the file.

How to place a file on salt master via salt-api

I want to place a file a file on salt-master via salt-api. I have configured salt-api using rest cherrypy and configured a custom hook for it. I wanted to explore the use-case where we can transfer the file first to salt-master and secondly distribute it to minions. I'm able to achieve the second part but not been able to post data file to the API.
Here is one way to do it using file.write execution module.
First login and save the token to a cookie file (I had to change eauth to ldap, auto didn't work for some reason):
curl -sSk http://localhost:8000/login \
-c ~/cookies.txt \
-H 'Accept: application/x-yaml' \
-d username=USERNAME\
-d password=PASSWORD \
-d eauth=auto
Now run a job to create a file on the salt-master (assuming your salt-master is also running a salt-minion):
curl -sSk http://localhost:8000 \
-b ~/cookies.txt \
-H 'Accept: application/x-yaml' \
-d client=local \
-d tgt='saltmaster' \
-d fun=file.write \
-d arg='/tmp/somefile.txt' \
-d arg='This is some example text
with newlines
A
B
C'
Note that the spacing used in your command will affect how the lines will show up in the file, with the example above is gives the most aesthetically pleasing result.

How to use pastebin from shell script?

Is it possible to use pastebin (may be via their "API" functionality) inside bash shell scripts? How do I send http-post? How do I get back the URL?
As pastebin.com closed their public api, I was looking for alternatives.
Sprunge is great. Usage:
<command> | curl -F 'sprunge=<-' http://sprunge.us
or, as I use it:
alias paste="curl -F 'sprunge=<-' http://sprunge.us"
<command> | paste
The documentation says that you need to submit a POST request to
http://pastebin.com/api_public.php
and the only mandatory parameter is paste_code, of type string is the paste that you want to make.
On success a new pastebin URL will be returned.
You can easily do this from your bash shell using the command curl.
curl uses the -d option to send the POST data to the specified URL.
Demo:
This demo will create a new paste with the code:
printf("Hello..I am Codaddict");
From your shell:
$ curl -d 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
http://pastebin.com/598VLDZp
$
Now if you see the URL http://pastebin.com/598VLDZp, you'll see my paste :)
Alternatively you can do it using the wget command which uses the option --post-data to sent POST values.
I've tried this command it works fine:
wget --post-data 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
Put the following in your .bashrc:
sprunge() {
if [[ $1 ]]; then
curl -F 'sprunge=<-' "http://sprunge.us" <"$1"
else
curl -F 'sprunge=<-' "http://sprunge.us"
fi
}
...and then you can run:
sprunge filename # post file to sprunge
...or...
some_command | sprunge # pipe output to sprunge
The API for posting to pastebin has changed, since posted by codaddict.
Details can be found at this link: https://pastebin.com/api
Example:
curl -d 'api_paste_code=printf("Hello..\n I am Codaddict");' \
-d 'api_dev_key=<get_your_own>' \
-d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
There are three essential fields as of now:
api_dev_key -> You need to create a login on pastebin.com in order to get that
api_option -> Format in which to post
api_paste_code -> Text you want to post
Two other answers (from circa 2014) point to http://sprunge.us, which is designed to be used like this...
curl --form 'sprunge=#yourfile.txt' sprunge.us
However, as of 2018, sprunge.us has a tendency to be overloaded and return 500 Internal Server Error to every request. For files up to at least 300 KB but not as high as 2.8 MB, I have had good luck with the very similar service at http://ix.io:
curl --form 'f:1=#yourfile.txt' ix.io
For files up to at least 2.8 MB (and maybe higher, I don't know), I've found the more highly polished https://transfer.sh. It recommends a slightly different and simpler command line, and requires https (it won't work without it):
curl --upload-file yourfile.txt https://transfer.sh
I have found that Sprunge is currently down, but dpaste.com has a simple API.
To post from STDIN
curl -s -F "content=<-" http://dpaste.com/api/v2/
from a file foo.txt
cat foo.txt | curl -s -F "content=<-" http://dpaste.com/api/v2/
to post a string
curl -s -F "content=string" http://dpaste.com/api/v2/
The response will be a plain text URL to the paste.
Nb: the trailing / in the URL http://dpaste.com/api/v2/ seems necessary
https://paste.c-net.org/ has a simpler API than all of them. Simply "POST" to it.
From the website:
Upload text using curl:
$ curl -s --data 'Hello World!' 'https://paste.c-net.org/'
Upload text using wget:
$ wget --quiet -O- --post-data='Hello World!' 'https://paste.c-net.org/'
Upload a file using curl:
$ curl --upload-file #'/tmp/file' 'https://paste.c-net.org/'
Upload a file using wget:
$ wget --quiet -O- --post-file='/tmp/file' 'https://paste.c-net.org/'
Upload the output of a command or script using curl:
$ ls / | curl --upload-file - 'https://paste.c-net.org/'
$ ./bin/hello_world | curl -s --data-binary #- 'https://paste.c-net.org/'
You can also simply use netcat. Unlike termbin, paste.c-net.org won't time out if your script takes more than 5 seconds to produce its output.
$ { sleep 10; ls /; } | nc termbin.com 9999
$ { sleep 10; ls /; } | nc paste.c-net.org 9999
https://paste.c-net.org/ExampleOne
Easiest way to post to pastebin
echo 'your message' | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key=<your_api_key>' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
Just change the <your_api_key> part and pipe whatever you want into it.
The sed invocations add the api_paste_code parameter to beginning of the message and add a newline at the end of each line so it can handle multiline input. The #- tells curl to read from stdin.
A Bash Function You Can Paste
For easy reuse, make it a bash function (copy and paste this into your terminal and set the API_KEY field appropriately:
pastebin () {
API_KEY='<your_api_key>'
if [ -z $1 ]
then
cat - | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
else
echo "$1" | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
fi
printf '\n'
}
You can run it with either:
pastebin 'your message'
or if you need to pipe a file into it:
cat your_file.txt | pastebin
To built upon Vishal's answer, pastebin has upgraded to only use HTTPS now:
curl -d 'api_paste_code=printf("Hello World");' \
-d 'api_dev_key=<your_key>' \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
You don't have to specify the -X POST parameter
Additional details can be found here:
https://pastebin.com/doc_api#1
Based on another answer on this page, I wrote the following script which reads from STDIN (or assumes output it piped into it).
This version allows for arbitrary data which is URI escaped (by jq).
#!/bin/bash
api_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
curl -d "api_paste_code=$(jq -sRr #uri)" \
-d "api_dev_key=$api_key" \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
echo # By default, there's no newline
I am a bit late to this post, but I created a little tool to help with this.
https://pasteshell.com/
Feel free to check it out and let me know what you think.
Thanks,

Resources