Is there any way I can copy data directly from the Dropbox servers to a UNIX server, without being root or having Dropbox software installed there?
Something like:
rsync -aP myusername#dropbox.com:somepath/ .
(The reason for wanting to do this is that the transfer speed between the UNIX server and the Dropbox server would be much faster since they are both on the backbone, than between my local machine and the UNIX server, which have a home broadband connection).
You can use wget to download the file from dropbox to the Server from the command line. some thing like
wget http://dl.dropbox.com/u/12345678/largefile.zip
If you are on a slow link and the download cuts off before all of the file is downloaded, you can resume download from wherever it stopped, by using the -c option with wget. Just do:
wget -c http://dl.dropbox.com/u/12345678/largefile.zip
Try rclone https://github.com/dropbox/dbxcli/issues/60#issuecomment-497713363
install
$ curl -OJN https://downloads.rclone.org/rclone-current-linux-amd64.zip
$ unzip rclone-current-linux-amd64.zip
$ cp rclone-v1.47.0-linux-amd64/rclone ~/bin/
config https://rclone.org/dropbox/
$ rclone config
2019/05/31 15:00:07 NOTICE: Config file "/home/roman/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> dropbox
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / A stackable unification remote, which can appear to merge the contents of several remotes
\ "union"
2 / Alias for a existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Dropbox
\ "dropbox"
9 / Encrypt/Decrypt a remote
\ "crypt"
10 / FTP Connection
\ "ftp"
11 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
12 / Google Drive
\ "drive"
13 / Hubic
\ "hubic"
14 / JottaCloud
\ "jottacloud"
15 / Koofr
\ "koofr"
16 / Local Disk
\ "local"
17 / Mega
\ "mega"
18 / Microsoft Azure Blob Storage
\ "azureblob"
19 / Microsoft OneDrive
\ "onedrive"
20 / OpenDrive
\ "opendrive"
21 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
22 / Pcloud
\ "pcloud"
23 / QingCloud Object Storage
\ "qingstor"
24 / SSH/SFTP Connection
\ "sftp"
25 / Webdav
\ "webdav"
26 / Yandex Disk
\ "yandex"
27 / http Connection
\ "http"
Storage> 8
** See help for dropbox backend at: https://rclone.org/dropbox/ **
Dropbox App Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Dropbox App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n>
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
--------------------
[dropbox]
type = dropbox
token = {"access_token":"<token>","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
dropbox dropbox
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
copy all files from dropbox https://rclone.org/docs/
$ rclone copy --dry-run dropbox:/ .
$ rclone copy dropbox:/ .
I don't believe you have SSH access to the Dropbox server. Think about it, that would mean the Dropbox server would have thousands if not millions of SSH accounts, just in the chance that a user would use it. Additionally, a default Unix user has access to read so much of the Unix OS and restricting it to just a few commands is somewhat of a big deal. Imagine if your Unix user was somehow able to see the data of another Unix user including their Dropbox data. Good idea though.
Edit: capitalization.
Related
When I need to copy a file from local server (server A) to remote server(server B) via SSH, using a user with enough privileges, I do this successfuly like below
localpath='/this/is/local/path/file1.txt'
remotepath='/this/is/remote/path/'
mypass='MyPassword123'
sshpass -p $mypass scp username#hostname:$localpath $remotepath
Now, I have to transfer a file from server A to server C with a user that doesn't have enough privileges to copy. Then once
I connected to Server C, I need to send su in order to be able to send commands like cd, ls, etc.
Manually, I access the server C via SSH like this:
[root#ServerA ~]# ssh username#hostname
You are trying to access a restricted zone. Only Authorized Users allowed.
Password:
Last login: Sat Jun 13 10:17:40 2020 from XXX.XXX.XXX.XXX
ServerC ~ $
ServerC ~ $ su
Password:
ServerC /home/myuser #
ServerC /home/myuser # cd /documents/backups/
ServerC /documents/backups #
At this moment myuser has superuser privileges and I can send commands.
Then, how can I automate the task to copy files from server A to server C with the need to send su once I'm connected to Server C?
I've tried so far doing like this:
sshpass -p $mypass ssh -t username#hostname "su -c \"cd /documents/backups/ && ls\""
it requests password for su and I'm able to send cd and ls but with this command, I'm not copying files from Server A to Server C, only semi-automating the access to Server C and sending the su in Server C.
Thanks in advance for any help.
UPDATE
# $TAR | ssh $username#$hostname "$COMMAND"
+ tar -cv -C /this/is/local/path/file1.txt .
+ ssh username#X.X.X.X 'set -x; rm -f /tmp/copy && mknod /tmp/copy p; su - <<< "su_password
set -x; tar -xv -C /this/is/remote/path/ . < /tmp/copy" & cat > /tmp/copy'
tar: /this/is/local/path/file1.txt: Cannot chdir: Not a directory
tar: Error is not recoverable: exiting now
You are trying to access a restricted zone. Only Authorized Users allowed.
Password:
+ rm -f /tmp/copy
+ mknod /tmp/copy p
+ su -
+ cat
Password:
Editorial note: the previous version of this answer used sudo, the current version uses su as requested in the question.
You could use tar and pipes, like so:
TAR="tar -cv -C $localpath ."
UNTAR="tar -xv -C $remotepath ."
PREPARE_PIPE="rm -f /tmp/copy && mknod /tmp/copy p"
NEWLINE=$'\n' # that's the easiest way to get a literal newline
ROOT_PASSWORD=rootpasswordverydangerous
COMMAND="set -x; $PREPARE_PIPE; su - <<< \"${ROOT_PASSWORD}${NEWLINE} set -x; $UNTAR < /tmp/copy\" & cat > /tmp/copy"
$TAR | ssh username#hostname "$COMMAND"
Explanation:
tar -c . archives the current directory into a single file. We aren't passing -f to tar, so that single file is standard output.
tar -x . extracts the content of a single tar archive file to the current directory. We aren't passing -f to tar, so that single file is standard input.
-C <path> tells tar to cd into <path> so that it will be the current directory in which files are copied from/to.
-v just tells tar to list the files tar archives/extracts, for debugging purposes.
Likewise, set -x is just to have bash to emit trace information, for debugging purposes.
So we're archiving $localpath into stdout, and piping it to ssh, which will pipe it to $COMMAND.
If there was a way to give su the password in the command line, we would have used something like:
$TAR | ssh ... su --password ${ROOT_PASSWORD} -c "$UNTAR"
and things would have been simple.
But su doesn't have that. su runs like a shell, reading from stdin. So it will first read the password, and once the password is read and su has established a root session, it reads commands from stdin. That's why we have su - <<< \"${ROOT_PASSWORD}${NEWLINE}${UNTAR}.
But now stdin is used by the password and command, so we can't use it as the archive. We could use another file descriptor, but I prefer not to, because then the solution can be more easily ported to work with sudo instead of su. sudo closes all file descriptors, and sudo -C 200 (only close file descriptors above 200) may not work (didn't work on my test machine).
if we went that direction, we would have used something like
$TAR | ssh ... 'exec 9<&2 && sudo -S <<< $mypass bash -c "$UNTAR <&9"'
Our next option is to do something like cat > /tmp/archive.tar in order to write the entire archive into a file, and then have something like $UNTAR < /tmp/archive.tar. But the archive may be huge and we may run out of disk space.
So the idea is to create a dedicated pipe - that's PREPARE_PIPE. Pipes don't save anything to disk, and don't store the entire stream in memory, so the reader and the writer have to work concurrently (you know, like with a real pipe).
So having redirected su's stdin from $ROOT_PASSWORD, we pull ssh's stdin into our pipe with cat > /tmp/copy, and in parallel (&) having $UNTAR read from the pipe (< /tmp/copy).
Notes:
You could also pass -z to both tar commands to pass it compressed, if your network is too slow.
tar will preserve the source's metadata, e.g. timestamps and ownership.
Passing $ROOT_PASSWORD to commands is not good practice, anyone who runs ps -ef can see the password. There are ways to pass the password to server C in a more secure way, I didn't include it in order to not further complicate this answer.
I would suggest asking the server's owner to install sudo, so that if the password is compromised via ps -ef, at least it's not the root password.
I'm using a local repository as a staging repo and would like to be able to clear the whole staging repo via REST. How can I delete the contents of the repo without deleting the repo itself?
Since I have a similar requirement in one of my environments I like to provide a possible solution approach.
It is assumed the JFrog Artifactory instance has a local repository called JFROG-ARTIFACTORY which holds the latest JFrog Artifactory Pro installation RPM(s). For listing and deleting I've created the following script:
#!/bin/bash
# The logged in user will be also the admin account for Artifactory REST API
A_ACCOUNT=$(who am i | cut -d " " -f 1)
LOCAL_REPO=$1
PASSWORD=$2
STAGE=$3
URL="example.com"
# Check if a stage were provided, if not set it to PROD
if [ -z "$STAGE" ]; then
STAGE="repository-prod"
fi
# Going to list all files within the local repository
# Doc: https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-FileList
curl --silent \
-u"${A_ACCOUNT}:${PASSWORD}" \
-i \
-X GET "https://${STAGE}.${URL}/artifactory/api/storage/${LOCAL_REPO}/?list&deep=1" \
-w "\n\n%{http_code}\n"
echo
# Going to delete all files in the local repository
# Doc: https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-DeleteItem
curl --silent \
-u"${A_ACCOUNT}:${PASSWORD}" \
-i \
-X DELETE "https://${STAGE}.${URL}/artifactory/${LOCAL_REPO}/" \
-w "\n\n%{http_code}\n"
echo
So after calling
./Scripts/deleteRepository.sh JFROG-ARTIFACTORY Pa\$\$w0rd! repository-dev
for the development instance, it listed me all files in the local repository called JFROG-ARTIFACTORY, the JFrog Artifactory Pro installation RPM(s), deleted them, but left the local repository itself.
You may change and enhance the script for your needs and have also a look into How can I completely remove artifacts from Artifactory?
This problem is not specific to rsync. If I touch a file named /media/KINGSTON/seventeen. then what is created is /media/KINGSTON/seventeen instead. Can someone explain why?
dmesg
. . .
ugen3.2: <Kingston DataTraveler 3.0> at usbus3
umass0 on uhub6
umass0: <Kingston DataTraveler 3.0, class 0/0, rev 2.10/1.10, addr 2> on usbus3
umass0: SCSI over Bulk-Only; quirks = 0x8100
umass0:5:0: Attached to scbus5
da0 at umass-sim0 bus 0 scbus5 target 0 lun 0
da0: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 485B39472CCAB171D76F0DF0
da0: 40.000MB/s transfers
da0: 118272MB (242221056 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
gpart show /dev/da0*
=> 63 242220993 da0 MBR (116G)
63 31041 - free - (15M)
31104 242189952 1 !12 [active] (115G)
I am backing up a cyrus-imap mailstore using rsync. Cyrus imap message file names are numbers followed by a dot (####.). When these message files are transferred using rsync on a FreeBSD-11.2 host the trailing dot is removed on the target file name (####. becomes ####). Is there some way to prevent this behaviour?
rsync \
--copy-links \
--no-group \
--no-perms \
--progress \
--protect-args \
--modify-window=1 \
--recursive \
--times \
--update \
--verbose \
./Documents/Personal/IMAP \
/media/KINGSTON/Documents/Personal/IMAP
It appears from further testing that this behaviour is dependent upon the destination. When copied from and to the system hdd the trailing dot appears in the target file name. When the target is a USB key then the dot disappears from the target.
A trailing dot or space is not permitted in a valid MS Windows file name. The transferred files have their names silently altered to meet this requirement when copied to a FAT formatted USB.
From Microsoft file naming conventions (https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file):
Do not end a file or directory name with a space or a period. Although
the underlying file system may support such names, the Windows shell
and user interface does not. However, it is acceptable to specify a
period as the first character of a name. For example, ".temp".
Has anyone written something like davcopy for Livelink? (davcopy works with SharePoint)
I have downloaded davcopy and it hangs when trying to use it with Livelink.
I've asked Open Text and their response is "There is not way to do this out of the box, it will requires writing a webservices application."
I'm not sure how to write a webservice application for livelink; so, before I explore that I was wondering if anyone had done an implementation of davcopy for Livelink.
I know about a command line application which is using MS powershell to do what you want (http://www.gatevillage.net/public/content-server-desktop-library-powershell-suite)
It wouldn't be too difficult to write something like this with Ruby or Perl. Both support WS/SOAP.
With which version of Livelink (or Content Server) do you work?
You can use the curl command line tool to upload, download or delete files in Livelink. It makes HTTP requests against CS REST API, which is available in CS 10.0 or newer.
For example, uploading a file "file.ext" to folder 8372 at http://server/instance/cs as Admin:
curl \
-F "type=144" \
-F "parent_id=8372" \
-F "name=file.ext" \
-F "file=#/path/to/file.ext" \
-u "Admin:password" \
-H "Expect:" \
http://server/instance/cs/api/v1/nodes
The "Expect" header has to be forced empty, because CS REST API does not support persistent connections, but curl would always enable them for this request.
I have a single vmware disk image file with vmdk extension
I am trying to mount this and explore all of the partitions (including hidden ones).
I've tried to follow several guides, such as : http://forums.opensuse.org/showthread.php/469942-mounting-virtual-box-machine-images-host
I'm able to mount the image using vdfuse
vdfuse -w -f windows.vmdk /mnt/
After this I can see one partition and an entire disk exposed
# ll /mnt/
total 41942016
-r-------- 1 te users 21474836480 Feb 28 14:16 EntireDisk
-r-------- 1 te users 1569718272 Feb 28 14:16 Partition1
Continuing with the guide I try to mount either EntireDisk or Partition1 using
mount -o loop,ro /mnt/Partition1 mnt2/
But that gives me the error 'mount: you must specify a filesystem type'
In trying to find the correct type I tried
dd if=/mnt/EntireDisk | file -
which outputs a ton of information but of note is:
/dev/stdin: x86 boot sector; partition 1: ....... FATs ....
So i tired to mount as a vfat but that gave me
mount: wrong fs type, bad option, bad superblock ...etc
What am I doing wrong?
For newer Linux systems, you can use guestmount to mount the third partition within a VMDK image:
guestmount -a xyz.vmdk -m /dev/sda3 --ro /mnt/vmdk
Alternatively, to autodetect and mount an image (less reliable), you can try:
guestmount -a xyz.vmdk -i --ro /mnt/vmdk
Do note that the flag --ro simply mounts the image as read-only; to mount the image as read-write, just replace it with the flag --rw.
Installation
guestmount is contained in following packages per distro:
Ubuntu: libguestfs-tools
OpenSuse: guestfs-tools
CentOS / Fedora: libguestfs-tools-c
Troubleshooting
error: could not create appliance through libvirt
$ guestmount -a file.vmdk -i --ro /mnt/guest
libguestfs: error: could not create appliance through libvirt.
Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct
Original error from libvirt: Cannot access backing file '/path/to/file.vmdk' of storage file '/tmp/libguestfssF6WKX/overlay1.qcow2' (as uid:107, gid:107): Permission denied [code=38 int1=13]
Solution: use LIBGUESTFS_BACKEND=direct, as suggested:
LIBGUESTFS_BACKEND=direct guestmount -a file.vmdk -i --ro /mnt/guest
fusermount: user has no write access to mountpoint
LIBGUESTFS_BACKEND=direct guestmount -a file.vmdk -i --ro /mnt/guest/
fusermount: user has no write access to mountpoint /mnt/guest
libguestfs: error: fuse_mount failed: /mnt/guest/, see error messages above
Solution: use sudo, or change file permissions on the mountpoint
You can also use qemu:
For .vdi disks
sudo modprobe nbd
sudo qemu-nbd -c /dev/nbd1 ./linux_box/VM/image.vdi
if they are not installed, you can install them (issuing this command in Ubuntu)
sudo apt install qemu-utils
and then mount it with:
mount /dev/nbd1p1 /mnt
For .vmdk disks
sudo modprobe nbd
sudo qemu-nbd -r -c /dev/nbd1 ./linux_box/VM/image.vmdk
notice that I use the option -r, that's because VMDK version 3 must be read only to be able to be mounted by qemu
and then I mount it with
mount /dev/nbd1p1 /mnt
I use nbd1, because nbd0 sometimes gives: 'mount: special device /dev/nbd0p1 does not exist'
For .ova disks
tar -tf image.ova
tar -xvf image.ova
The above will extract the .vmdk disk and then mount it.
Install affuse, then mount using it.
affuse /path/file.vmdk /mnt/vmdk
The raw disk image is now found under /mnt/vmdk.
Check its sector size:
fdisk -l /mnt/vmdk/file.vmdk.raw
# example
Disk file.vmdk.raw: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000da525
Device Boot Start End Sectors Size Id Type
/mnt/vmdk/file.vmdk.raw1 * 2048 41943039 41940992 20G 83 Linux
Multiply sector size and start sector. In the example it would be 2048*512:
echo '2048*512' | bc
1048576
Mount the raw file using that offset:
mount -o ro,loop,offset=1048576 /mnt/vmdk/file.raw /mnt/vmdisk
The disk should now be mounted and readable on /mnt/vmdisk.
Here is an answer from commandlinefu.com that worked for me:
kpartx -av <image-flat.vmdk>; mount -o /dev/mapper/loop0p1 /mnt/vmdk
You can also activate LVM volumes in the image by running
vgchange -a y
and then you can mount the LV inside the image.
To unmount the image, umount the partition/LV, deactivate the VG for the image
vgchange -a n <volume_group>
then run
kpartx -dv <image-flad.vmdk>
to remove the partition mappings.
You can take a look in this article for a download link for VMware Virtual Disk Development Kit (VDDK). Once downloaded and installed:
vmware-mount -p path_to_vmdk will show the partitions inside the VMDK file. For example:
Nr Start Size Type Id Sytem
-- ---------- ---------- ---- -- ------------------------
1 2048 461371392 BIOS 83 Linux
Then just do:
sudo vmware-mount path_to_vmdk 1 /mnt/mount_point
I tried guestmount, but it is very, very slow. Underneath it creates a virtual machine, uses KVM and so on. Crazy stuff, slow as hell.
Have you got the software package for ntfs?
Try
apt-get install ntfs-3g
on debian based systems.