Bash sftp expect Script Does Not Continue After Retrieving Multiple Files - sftp

I need to script out a way to FTP files from a remote server. I cannot use keys, so the login needs to be "interactive". I felt that expect was the way to go.
I have the following expect code in a bash script:
expect <<EOD
spawn sftp $sftp_user#$sftp_svr
expect "password:"
send "$sftp_pw\r"
expect "sftp> "
send "mget $getfile_path$getfile_name $savefile_path\r"
expect "sftp> "
send "bye\r"
EOD
The snippet works fine when it only retrieves one file. However, if I need to retrieve more than one file the final "expect sftp>" is never acted on.
Working:
[ed#svr1 ~]$ ./sftp_expect.sh
spawn sftp user#svr2
user#svr2's password:
Connected to svr2.
sftp> mget /data/user/upload/lastcheck1.txt /home/ed/report_received/
Fetching /data/user/upload/lastcheck1.txt to /home/ed/report_received/lastcheck1.txt
/data/user/upload/lastcheck1.txt 100% 56 26.9KB/s 00:00
sftp>
[ed#svr1 ~]$
Not Working:
[ed#svr1 ~]$ ./sftp_expect.sh
spawn sftp user#svr2
user#svr2's password:
Connected to svr2.
sftp> mget /data/user/upload/lastcheck*.txt /home/ed/report_received/
Fetching /data/user/upload/lastcheck1.txt to /home/ed/report_received/lastcheck1.txt
/data/user/upload/lastcheck1.txt 100% 56 26.5KB/s 00:00
Fetching /data/user/upload/lastcheck2.txt to /home/ed/report_received/lastcheck2.txt
/data/hf_user/upload/lastcheck2.txt 100% 56 19.3KB/s 00:00
Fetching /data/user/upload/lastcheck_svr2.txt to /home/ed/report_received/lastcheck_svr2.txt
/data/user/upload/lastcheck_svr2.txt 100% 56 31.9KB/s 00:00
sftp>
^C
[ed#svr1 ~]$
The script gets the files, expect gets "what it's expecting" but it does not continue on:
expect: does "mget /data/user/upload/lastcheck*.txt /home/ed/report_received/\r\nFetching /data/user/upload/lastcheck1.txt to /home/ed/report_received/lastcheck1.txt\r\n\r/data/user/upload/lastcheck1.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck1.txt 100% 56 24.5KB/s 00:00 \r\nFetching /data/user/upload/lastcheck2.txt to /home/ed/report_received/lastcheck2.txt\r\n\r/data/user/upload/lastcheck2.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck2.txt 100% 56 38.3KB/s 00:00 \r\nFetching /data/user/upload/lastcheck_svr2.txt to /home/ed/report_received/lastcheck_svr2.txt\r\n\r/data/user/upload/lastcheck_svr2.txt 0% 0 0.0KB/s --:-- ETA" (spawn_id exp6) match glob pattern "sftp> "? no
/data/user/upload/lastcheck_svr2.txt 100% 56 25.7KB/s 00:00
expect: does "mget /data/user/upload/lastcheck*.txt /home/ed/report_received/\r\nFetching /data/user/upload/lastcheck1.txt to /home/ed/report_received/lastcheck1.txt\r\n\r/data/user/upload/lastcheck1.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck1.txt 100% 56 24.5KB/s 00:00 \r\nFetching /data/user/upload/lastcheck2.txt to /home/ed/report_received/lastcheck2.txt\r\n\r/data/user/upload/lastcheck2.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck2.txt 100% 56 38.3KB/s 00:00 \r\nFetching /data/user/upload/lastcheck_svr2.txt to /home/ed/report_received/lastcheck_svr2.txt\r\n\r/data/user/upload/lastcheck_svr2.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck_svr2.txt 100% 56 25.7KB/s 00:00 \r\n" (spawn_id exp6) match glob pattern "sftp> "? no
sftp>
expect: does "mget /data/user/upload/lastcheck*.txt /home/ed/report_received/\r\nFetching /data/user/upload/lastcheck1.txt to /home/ed/report_received/lastcheck1.txt\r\n\r/data/user/upload/lastcheck1.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck1.txt 100% 56 24.5KB/s 00:00 \r\nFetching /data/user/upload/lastcheck2.txt to /home/ed/report_received/lastcheck2.txt\r\n\r/data/user/upload/lastcheck2.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck2.txt 100% 56 38.3KB/s 00:00 \r\nFetching /data/user/upload/lastcheck_svr2.txt to /home/ed/report_received/lastcheck_svr2.txt\r\n\r/data/user/upload/lastcheck_svr2.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck_svr2.txt 100% 56 25.7KB/s 00:00 \r\nsftp> " (spawn_id exp6) match glob pattern "sftp> "? yes
expect: set expect_out(0,string) "sftp> "
expect: set expect_out(spawn_id) "exp6"
expect: set expect_out(buffer) "mget /data/user/upload/lastcheck*.txt /home/ed/report_received/\r\nFetching /data/user/upload/lastcheck1.txt to /home/ed/report_received/lastcheck1.txt\r\n\r/data/user/upload/lastcheck1.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck1.txt 100% 56 24.5KB/s 00:00 \r\nFetching /data/user/upload/lastcheck2.txt to /home/ed/report_received/lastcheck2.txt\r\n\r/data/user/upload/lastcheck2.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck2.txt 100% 56 38.3KB/s 00:00 \r\nFetching /data/user/upload/lastcheck_svr2.txt to /home/ed/report_received/lastcheck_svr2.txt\r\n\r/data/user/upload/lastcheck_svr2.txt 0% 0 0.0KB/s --:-- ETA\r/data/user/upload/lastcheck_svr2.txt 100% 56 25.7KB/s 00:00 \r\nsftp> "
send: sending "bye\r" to { exp6 }
^C
[ed#svr2 ~]$
If anyone has an idea what is holding up the expect piece I would very much appreciate it.
Ed.

expect <<EOD
set timeout -1
spawn sftp $sftp_user#$sftp_svr
expect "password:"
send "$sftp_pw\r"
expect "sftp> "
send "mget $getfile_path$getfile_name $savefile_path\r"
expect "sftp> "
send "bye\r"
EOD

Related

How do I unpack a project.tar from *Paw for Teams*?

Paw for Teams dashboard lets me download a .tar file of a team project:
Your project data is safely stored on the Amazon S3 storage service. In case you need a data dump for backup purposes you can download a tar file here (data is contained inside the hidden Git directory in data/.git).
This behavior seems different than the 2016 behavior described here.
I expected to be able to be able to tar xvf the file on my Mac, but it doesn't recognize the format:
~/Downloads> tar xvf Spikes.tar
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
Looking at the tar file, it seems to be uncompressed .git data:
00000000: 8004 95f0 0400 0000 0000 007d 9428 8c05 ...........}.(..
00000010: 6669 6c65 7394 7d94 288c 0448 4541 4494 files.}.(..HEAD.
00000020: 4317 7265 663a 2072 6566 732f 6865 6164 C.ref: refs/head
00000030: 732f 6d61 7374 6572 0a94 8c06 636f 6e66 s/master....conf
00000040: 6967 9443 425b 636f 7265 5d0a 0962 6172 ig.CB[core]..bar
00000050: 6520 3d20 7472 7565 0a09 7265 706f 7369 e = true..reposi
00000060: 746f 7279 666f 726d 6174 7665 7273 696f toryformatversio
00000070: 6e20 3d20 300a 0966 696c 656d 6f64 6520 n = 0..filemode
00000080: 3d20 7472 7565 0a94 8c0b 6465 7363 7269 = true....descri
00000090: 7074 696f 6e94 4349 556e 6e61 6d65 6420 ption.CIUnnamed
000000a0: 7265 706f 7369 746f 7279 3b20 6564 6974 repository; edit
But I can't seem to find the format or tools to extract it. Any ideas?
Micha's comment answers the question: Paw backups are now in a proprietary format whose main purpose is to be transparent about what data is stored.

Ffmpeg/Fluent-Ffmpeg: Output File Does Not Reflect Input Parameters

Use Case
When a user records an audio file and uploads to firebase storage, run a trigger that crops a clip of that file and saves it to a "preview" directory.
Problem
When the file arrives on the server, it is just dead air before the start time (10 seconds) and then doesn't stop after the duration of 10 seconds is exceeded. Whatever I put in the "-t" and "-ss" arguments seems to have no impact on the output file - not cropping correctly.
Environment
mac client / firebase storage
node v8.1.0
ffmpeg v3.2.2
fluent-ffmpeg v2.1.2
Node Code
var command = new ffmpeg({ source: tempFilePath, timeout: 0 })
.setFfmpegPath(ffmpegPath)
.setFfprobePath(ffprobePath)
.inputOption('-t', '10')
.inputOption('-ss', '10')
.outputOption('-acodec', 'copy')
.on('start', function(commandLine) {
console.log('Spawned Ffmpeg with command: ', commandLine);
})
.on('end', function() {
console.log('Preview file cropping done successfully');
})
.on('error', function(err, stdout, stderr) {
var build = err;
if (!stdout === '') { build = build + '\n' + stdout; }
build = build + '\n' + stderr;
console.log(build);
}).save(tempFilePath);
Ffmpeg Command
Spawned Ffmpeg with command: ffmpeg -t 10 -i
/tmp/2EB100B0-6C1E-4D51-9087-764CA653AFC720170711232034.aac -y -acodec
aac /tmp/2EB100B0-6C1E-4D51-9087-764CA653AFC720170711232034.aac
Console Output
$ ffmpeg -t 10 -i E506FE2E-DC81-4DD8-B751-211F7FD9A25820170816160502.aac -y -acodec aac output.aac -loglevel debug
ffmpeg version 3.2.2 Copyright (c) 2000-2016 the FFmpeg developers
built with llvm-gcc 4.2.1 (LLVM build 2336.11.00)
configuration: --prefix=/Volumes/Ramdisk/sw --enable-gpl --enable-pthreads --enable-version3 --enable-libspeex --enable-libvpx --disable-decoder=libvpx --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-avfilter --enable-libopencore_amrwb --enable-libopencore_amrnb --enable-filters --enable-libgsm --enable-libvidstab --enable-libx265 --disable-doc --arch=x86_64 --enable-runtime-cpudetect
libavutil 55. 34.100 / 55. 34.100
libavcodec 57. 64.101 / 57. 64.101
libavformat 57. 56.100 / 57. 56.100
libavdevice 57. 1.100 / 57. 1.100
libavfilter 6. 65.100 / 6. 65.100
libswscale 4. 2.100 / 4. 2.100
libswresample 2. 3.100 / 2. 3.100
libpostproc 54. 1.100 / 54. 1.100
Splitting the commandline.
Reading option '-t' ... matched as option 't' (record or transcode "duration" seconds of audio/video) with argument '10'.
Reading option '-i' ... matched as input url with argument 'E506FE2E-DC81-4DD8-B751-211F7FD9A25820170816160502.aac'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'aac'.
Reading option 'output.aac' ... matched as output url.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option loglevel (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url E506FE2E-DC81-4DD8-B751-211F7FD9A25820170816160502.aac.
Applying option t (record or transcode "duration" seconds of audio/video) with argument 10.
Successfully parsed a group of options.
Opening an input file: E506FE2E-DC81-4DD8-B751-211F7FD9A25820170816160502.aac.
[file # 0x7fe42f700420] Setting default whitelist 'file,crypto'
[aac # 0x7fe431000000] Format aac probed with size=2048 and score=51
[aac # 0x7fe431000000] Before avformat_find_stream_info() pos: 0 bytes read:32928 seeks:4 nb_streams:1
[aac # 0x7fe431000000] All info found
[aac # 0x7fe431000000] Estimating duration from bitrate, this may be inaccurate
[aac # 0x7fe431000000] After avformat_find_stream_info() pos: 37888 bytes read:98464 seeks:4 frames:50
Input #0, aac, from 'E506FE2E-DC81-4DD8-B751-211F7FD9A25820170816160502.aac':
Duration: 00:01:24.28, bitrate: 335 kb/s
Stream #0:0, 50, 1/28224000: Audio: aac (LC), 44100 Hz, stereo, fltp, 335 kb/s
Successfully opened the file.
Parsing a group of options: output url output.aac.
Applying option acodec (force audio codec ('copy' to copy stream)) with argument aac.
Successfully parsed a group of options.
Opening an output file: output.aac.
[file # 0x7fe42f7016e0] Setting default whitelist 'file,crypto'
Successfully opened the file.
detected 4 logical cores
[graph 0 input from stream 0:0 # 0x7fe42f701880] Setting 'time_base' to value '1/44100'
[graph 0 input from stream 0:0 # 0x7fe42f701880] Setting 'sample_rate' to value '44100'
[graph 0 input from stream 0:0 # 0x7fe42f701880] Setting 'sample_fmt' to value 'fltp'
[graph 0 input from stream 0:0 # 0x7fe42f701880] Setting 'channel_layout' to value '0x3'
[graph 0 input from stream 0:0 # 0x7fe42f701880] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:0x3
[audio format for output stream 0:0 # 0x7fe42f702060] Setting 'sample_fmts' to value 'fltp'
[audio format for output stream 0:0 # 0x7fe42f702060] Setting 'sample_rates' to value '96000|88200|64000|48000|44100|32000|24000|22050|16000|12000|11025|8000|7350'
[AVFilterGraph # 0x7fe42f6101a0] query_formats: 5 queried, 12 merged, 0 already done, 0 delayed
Output #0, adts, to 'output.aac':
Metadata:
encoder : Lavf57.56.100
Stream #0:0, 0, 1/44100: Audio: aac (LC), 44100 Hz, stereo, fltp, delay 1024, 128 kb/s
Metadata:
encoder : Lavc57.64.101 aac
Stream mapping:
Stream #0:0 -> #0:0 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
Last message repeated 1 times
[output stream 0:0 # 0x7fe42f701ee0] EOF on sink link output stream 0:0:default.
No more output streams to write to, finishing.
[aac # 0x7fe431001200] Trying to remove 344 more samples than there are in the queue
size= 160kB time=00:00:10.00 bitrate= 130.9kbits/s speed=20.2x
video:0kB audio:157kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.881580%
Input file #0 (E506FE2E-DC81-4DD8-B751-211F7FD9A25820170816160502.aac):
Input stream #0:0 (audio): 432 packets read (398208 bytes); 432 frames decoded (442368 samples);
Total: 432 packets (398208 bytes) demuxed
Output file #0 (output.aac):
Output stream #0:0 (audio): 431 frames encoded (441000 samples); 432 packets muxed (160716 bytes);
Total: 432 packets (160716 bytes) muxed
432 frames successfully decoded, 0 decoding errors
[AVIOContext # 0x7fe42f611040] Statistics: 0 seeks, 432 writeouts
[aac # 0x7fe431001200] Qavg: 4508.253
[AVIOContext # 0x7fe42f7004c0] Statistics: 458912 bytes read, 4 seeks

Angstrom eMMC root remount

I have an I.MX6 embedded system with an Angstrom Linux.
By adding the kernel parameter init=/bin/sh to the boot arguments I have now root access to the system. Now I want to add/change a user.
But the root is mounted as read only.
cat /etc/fstab
rootfs / squashfs defaults 0 0
proc /proc proc defaults 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
tmpfs /tmp tmpfs defaults 0 0
/dev/mmcblk0p1 /appconfig ext2 defaults 0 0
/dev/mmcblk0p2 /media/kernel_1 ext2 noauto 0 0
/dev/mmcblk0p3 /media/kernel_2 ext2 noauto 0 0
/dev/mmcblk0p5 /media/rootfs_b f2fs noauto 0 0
/dev/mmcblk0p6 /media/rootfs_1 squashfs noauto,nofail,ro 0 0
/dev/mmcblk0p7 /media/rootfs_2 squashfs noauto,nofail,ro 0 0
/dev/mmcblk0p8 /var f2fs defaults 0 0
There are two kernels, actual kernel_1 with rootfs_1 is running.
I tried to use mount -o remount,rw / but this isn't working.
The command mount isn't working ether:
mount: failed to read mtab: No such file or directory
[ 1.500600] mmc0: SDHCI controller on 2198000.usdhc [2198000.usdhc] using ADMA
[ 1.541992] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.560080] NET: Registered protocol family 10
[ 1.569834] sit: IPv6 over IPv4 tunneling driver
[ 1.581656] NET: Registered protocol family 17
[ 1.587361] Key type dns_resolver registered
[ 1.596955] ThumbEE CPU extension supported.
[ 1.601296] Registering SWP/SWPB emulation handler
[ 1.607158] mmc0: MAN_BKOPS_EN bit is not set
[ 1.614429] registered taskstats version 1
[ 1.630641] mmc0: new DDR MMC card at address 0001
[ 1.638019] Key type encrypted registered
[ 1.645244] mmcblk0: mmc0:0001 P1XXXX 3.60 GiB
[ 1.652696] input: gpio_buttons#0 as /devices/soc0/gpio_buttons#0/input/input0
[ 1.662385] mmcblk0boot0: mmc0:0001 P1XXXX partition 1 2.00 MiB
[ 1.668707] snvs_rtc 20cc034.snvs-rtc-lp: setting system clock to 1970-01-01 00:04:09 UTC (249)
[ 1.677753] mmcblk0boot1: mmc0:0001 P1XXXX partition 2 2.00 MiB
[ 1.700312] mmcblk0: p1 p2 p3 p4 < p5 p6 p7 p8 >
[ 1.806836] VFS: Mounted root (squashfs filesystem) readonly on device 179:7.
[ 1.818583] devtmpfs: mounted
[ 1.821894] Freeing unused kernel memory: 348K (c069f000 - c06f6000)
So I want to remount /dev/mmcblk0p6 (current root) as read-write.
Any idea?

cat /usr/share/dict/words > words.txt "No Space Left On Device"

I am trying to use that dictionary file on Unix but every time I use the command:
cat /usr/share/dict/words > words.txt
I get the message "No Space Left On Device". Has anyone else gotten this message and figured out a solution? I have searched around and haven't found anything.
I am working on a school server and I typed df -H to find out that I haven't even come close to using my available space so there has to be something else going on.
Output of df command--->
nfs.cs.fsu.edu:/home/misc/guest 11G 3.6G 6.3G 37% /home/guest
nfs.cs.fsu.edu:/home/majors 146G 139G 0 100% /home/majors
/dev/sda2 40631988 1703356 36831348 5% /
/dev/mapper/vg0-lv01 203147960 22470992 170191208 12%
/scratch4 /dev/sda7 4061540 74260 3777636 2% /tmp
/dev/mapper/vg0-lv00 314879212 251729436 46898256 85% /backup
/dev/sda6 10153988 839452 8790420 9% /var
/dev/sda5 20315812 10550172 8717000 55% /usr
/dev/sda1 101086 37382 58485 39% /boot tmpfs 4088804 0 4088804 0% /dev/shm
nfs.cs.fsu.edu:/home/faculty 442383104 391954976 27655936 94% /home/faculty
nfs.cs.fsu.edu:/home/grads 121892608 84729120 30892608 74% /home/grads
nfs.cs.fsu.edu:/home/courses 75325152 62762560 8689952 88% /home/courses
nfs.cs.fsu.edu:/home/misc/research 18283264 8096928 9242784 47% /home/research
nfs.cs.fsu.edu:/home/misc/contest 15481856 892352 13803072 7% /home/contest
nfs.cs.fsu.edu:/home/staff 14982144 4206688 10002176 30% /home/staff
nfs.cs.fsu.edu:/home/misc/guest 10157376 3507552 6125664 37% /home/guest
nfs.cs.fsu.edu:/home/majors 142203584 134866176 0 100% /home/majors
nfs.cs.fsu.edu:/home/class 30472192 2391424 26508096 9% /home/class
nfs.cs.fsu.edu:/home/sait 49580256 8202944 38818144 18% /home/sait
nfs.cs.fsu.edu:/home/system 298754432 238142048 45192416 85% /home/system
So,your claim that your hard disk is empty enough to accommodate those list of dictionary words was not true!
See,the output of df in second line says
nfs.cs.fsu.edu:/home/majors 146G 139G 0 100% /home/majors # 0 GB of space available...
that you have 100% of /home/majors partition filled,there is 0 GB of space left,against your thought!!!
The only step in such a case is to delete some of the files which are unnecessary from /home/majors partition!
Using df -H . will show the free disk space on the partition hosting the current directory, and makes easier to see where you are trying to write data.
Your df command shows that there's free space, but not on the partition you are trying to write to.

meaning of kannel status

we have configured kannel, and the status look like
SMS: received 0 (0 queued), sent 15133 (0 queued), store size 0
SMS: inbound (0.00,0.00,0.00) msg/sec, outbound (3.08,15.23,0.02) msg/sec
DLR: received 14232, sent 0
DLR: inbound (11.45,5.64,0.02) msg/sec, outbound (0.00,0.00,0.00) msg/sec
DLR: 980 queued, using internal storage
Box connections:
smsbox:vsmsc, IP 127.0.0.1 (0 queued), (on-line 8d 21h 38m 41s)
smsc1[smsc1] SMPP:xxxxx.xxxxx.com:2775/2775:xxxxx:SMPP (online 769120s, rcvd: sms 0 (0.00,0.00,0.00) / dlr 1 (0.00,0.00,0.00), sent: sms 1 (0.00,0.00,0.00) / dlr 0 (0.00,0.00,0.00), failed 0, queued 0 msgs)
in the DLR inbound, it showing (11.45,5.64,0.02) msg/sec. There is 3 value inside (). What is the meaning of each?
thanks
Average for the last minute;
Average for the last 5 minutes;
Average for all of runtime.

Resources