Libvirt Domain.InterfaceStats network io i/o always zero - xen

Hello,I use function "Domain.InterfaceStats" to get network i/o statistics,but always get things like these:vif1.0 rx_bytes 0 vif1.0 rx_packets 0 vif1.0 rx_errs 0 vif1.0 rx_drop 0
vif1.0 tx_bytes 0 vif1.0 tx_packets 0 vif1.0 tx_errs 0 vif1.0 tx_drop 0. and it seems statistics is always zero. How can I fix it?
In addition,I use HVM.and xen version is 4.2, libvirt version is 1.1.2.and network is set as default just bridge.
Thank you.

if the vm system is window, maybe install gplpv driver can be helpful.

Related

Strange "attempt to access beyond end of device", where to look?

I am running a few Xen servers on my company network. Recently, on one of them, I have been trying to rsync (on the Dom0 console) a big server image from another machine, but every time run in to a system crash after somewhere between 30 and 100 GB. The syslog and kernel log show me something like this:
Sep 12 16:41:19 ampxen1 kernel: [ 1730.917516] attempt to access beyond end of device
Sep 12 16:41:19 ampxen1 kernel: [ 1730.917518] dm-1: rw=1, want=8878402463988083936, limit=3759505408
Sep 12 16:41:19 ampxen1 kernel: [ 1730.917520] EXT4-fs warning (device dm-1): ext4_end_bio:323: I/O error 10 writing to inode 33030164 (offset 47354740736 size 5881856 starting block 1109800307998510491)
...continuing with several hundred thousands of similar lines per second, eventually making the machine unreachable. The very high number of the starting block of the EXT4-write operation (that's 10^18 or the exabyte range) is clearly what to look at, but I am unable to find any mention of what could be the cause.
The server is based on ubuntu-18.04.03, standard xen install from the repositories. Storage is two 2TB disks in RAID1, configured as seen below, EXT4 filesystem on the large partition used for our server images. I have checked the disks with smartctl and the file system(s) with e2fsck, for what it's worth. It seems to be a file system issue, but I am wondering whether the xen kernel could be involved. Any ideas of what to look for would be appreciated!
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 500G 0 loop
sda 8:0 0 1,8T 0 disk
├─sda1 8:1 0 476M 0 part /boot/efi
├─sda2 8:2 0 1,8T 0 part
│ └─md0 9:0 0 1,8T 0 raid1
│ ├─ampxen1.0-ampxen1.dom0 253:0 0 23,3G 0 lvm /
│ └─ampxen1.0-ampxen1.vms0 253:1 0 1,8T 0 lvm /srv/vms0
└─sda3 8:3 0 46,5G 0 part [SWAP]
sdb 8:16 0 1,8T 0 disk
├─sdb1 8:17 0 476M 0 part
├─sdb2 8:18 0 1,8T 0 part
│ └─md0 9:0 0 1,8T 0 raid1
│ ├─ampxen1.0-ampxen1.dom0 253:0 0 23,3G 0 lvm /
│ └─ampxen1.0-ampxen1.vms0 253:1 0 1,8T 0 lvm /srv/vms0
└─sdb3 8:19 0 46,5G 0 part [SWAP]
I finally figured out that the problem was something as trivial as a faulty RAM block – running a memtest showed lots of errors on one of the four 16GB blocks. It seems that memory was only maxed out exactly when copying large files, while my existing virtual servers on the machine were running just fine at all other times.

ipvsadm -L -n suddenly showing no active connections

I have a very odd problem in a proxy cluster of four Squid proxies:
One of the machine is the master. The mater is running ldirectord which is checking the availability of all four machines, distributing new client connections.
All over a sudden, after years of operation I'm encountering this problem:
1) The machine serving the master role is not being assigned new connections, old connections are served until a new proxy is assigned to the clients.
2) The other machines are still processing requests, taking over the clients from the master (so far, so good)
3) "ipvsadm -L -n" shows ever-decreasing ActiveConn and InActConn values.
Once I migrate the master role to another machine, "ipvsadm -L -n" is showing lots of active and inactive connections, until after about an hour the same thing happens on the new master.
Datapoint: This happened again this afternoon, and now "ipvsadm -L -n" shows:
TCP 141.42.1.215:8080 wlc persistent 1800
-> 141.42.1.216:8080 Route 1 98 0
-> 141.42.1.217:8080 Route 1 135 0
-> 141.42.1.218:8080 Route 1 1 0
-> 141.42.1.219:8080 Route 1 2 0
No change in the numbers quite some time now.
Some more stats (ipvsadm -L --stats -n):
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 141.42.1.215:8080 1990351 87945600 0 13781M 0
-> 141.42.1.216:8080 561980 21850870 0 2828M 0
-> 141.42.1.217:8080 467499 23407969 0 3960M 0
-> 141.42.1.218:8080 439794 19364749 0 2659M 0
-> 141.42.1.219:8080 521378 23340673 0 4335M 0
Value for "Conns" is constant now for all realservers and the virtual server now. Traffic is still flowing (InPkts increasing).
I examined the output of "ipvsadm -L -n -c" and found:
25 FIN_WAIT
534 NONE
977 ESTABLISHED
Then I waited a minute and got:
21 FIN_WAIT
515 NONE
939 ESTABLISHED
It turns out that a local bird installation was injecting router for the IP of the virtual server and thus taking precedence over ARP.

Determine Which Hard Drive the Database-Files/Transaction-Files are Stored

I have an old Sybase server whose database is acting up. I have tried rebuilding the file-system and the database file. But the problem returns. I want to replace the hard drive that the database-files and transaction-files are stored. I want to determine exactly which hard drive it is because I am not familiar with Unix. Moreover, I also want to see if those files are stored in the same hard drive as the operating system or not; if they are, I will need to re-install the operating system as well as restoring the database to the new hard drive. Obviously this will be better if the database-files and the transaction-files are not in the same hard drive as the operating system. Please help me to determine these two things.
So far, I have found these:
(1) I use sp_helpdb command and find that the database files and the transaction files are stored in these logical devices:
sybdbs
syblogs
master
sybdbs2
(2) I use sp_helpdevice command to look into those 4 logical devices shown above, and find that those logical devices are in these physical devices:
/dev/rdsk/c0t0d0s1
/dev/rdsk/c0t3d0s4
d_master
/dev/rdsk/sybdbs2
(3) When I use sp_helpdevice to show all the physical devices, I see this:
device_name physical_name description status cntrltype device_number low high
------------------ ------------------------------------------- ------------------------------------------------ ------ --------- ------------- -------- --------
historydump /export/home/syb11.dump/history.dump disk, dump device 16 2 0 0 0
isproddump /export/home/syb11.dump/isprod.dump disk, dump device 16 2 0 0 0
istestdump /export/home/syb11.dump/istest.dump disk, dump device 16 2 0 0 0
master d_master special, physical disk, 100.00 MB 2 0 0 0 51199
masterdump /export/home/syb11.dump/master.dump disk, dump device 16 2 0 0 0
modeldump /export/home/syb11.dump/model.dump disk, dump device 16 2 0 0 0
prodtestdump /export/home/syb11.dump/prodtest.dump disk, dump device 16 2 0 0 0
sybdbs /dev/rdsk/c0t0d0s1 special, default disk, physical disk, 2000.00 MB 3 0 3 50331648 51355647
sybdbs2 /dev/rdsk/sybdbs2 special, physical disk, 1.00 MB 2 0 5 83886080 83886591
syblogs /dev/rdsk/c0t3d0s4 special, physical disk, 850.00 MB 2 0 4 67108864 67544063
sybscurty /dev/rdsk/c0t3d0s5 special, physical disk, 100.00 MB 2 0 2 33554432 33605631
sybsecuritydump /export/home/syb11.dump/sybsecurity.dump disk, dump device 16 2 0 0 0
sybsystemprocsdump /export/home/syb11.dump/sybsystemprocs.dump disk, dump device 16 2 0 0 0
sysprocsdev /dev/rdsk/c0t0d0s4 special, physical disk, 100.00 MB 2 0 1 16777216 16828415
tapedump1 /dev/rmt4 tape, 625 MB, dump device 16 3 0 0 20000
tapedump2 /dev/rst0 disk, dump device 16 2 0 0 20000
uniface724dump /export/home/syb11.dump/uniface724.dump disk, dump device 16 2 0 0 0
uniface7dump /export/home/syb11.dump/uniface7.dump disk, dump device 16 2 0 0 0
(4) I want to know more about those physical devices. I use the df command to examine them:
df -k /dev/rdsk/c0t0d0s1
df -k /dev/rdsk/c0t3d0s4
df -k d_master
df -k /dev/rdsk/sybdbs2
The df command complains that the first three devices are “not a block device, directory or mounted resource”.
On the other hand, the df command shows the following info for the last device:
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t3d0s0 576558 371019 147889 71% /
In any case, this doesn’t tell me which drive(s) those devices are on.
(5) When I use the mount command, I see this:
/ on /dev/dsk/c0t3d0s0 read/write/setuid on Mon Jul 6 11:10:46 2015
/usr on /dev/dsk/c0t3d0s6 read/write/setuid on Mon Jul 6 11:10:46 2015
/proc on /proc read/write/setuid on Mon Jul 6 11:10:46 2015
/dev/fd on fd read/write/setuid on Mon Jul 6 11:10:46 2015
/tmp on swap read/write on Mon Jul 6 11:10:49 2015
/export on /dev/dsk/c0t3d0s7 setuid/read/write on Mon Jul 6 11:10:49 2015
/freespace on /dev/dsk/c0t0d0s5 setuid/read/write on Mon Jul 6 11:10:49 2015
/sybase on /dev/dsk/c0t0d0s0 setuid/read/write on Mon Jul 6 11:10:49 2015
/usr/openwin on /dev/dsk/c0t3d0s3 setuid/read/write on Mon Jul 6 11:10:49 2015
I cannot figure out the connection between the mounted devices above to the physical devices for the database-files and the transaction-files. I also cannot link the mounted devices above to the hard drives shown in the next section.
(6) When I use the cat /etc/vfstab command, I see these:
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
/proc - /proc proc - no -
fd - /dev/fd fd - no -
swap - /tmp tmpfs - yes -
/dev/dsk/c0t3d0s0 /dev/rdsk/c0t3d0s0 / ufs 1 no -
/dev/dsk/c0t3d0s6 /dev/rdsk/c0t3d0s6 /usr ufs 1 no -
/dev/dsk/c0t3d0s7 /dev/rdsk/c0t3d0s7 /export ufs 2 yes -
/dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /freespace ufs 2 yes -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 /sybase ufs 2 yes -
/dev/dsk/c0t3d0s3 /dev/rdsk/c0t3d0s3 /usr/openwin ufs 2 yes -
/dev/dsk/c0t3d0s1 - - swap - no -
# The following lines have been commented-out to allow Sybase to access these
# partitions and Raw Partitions. Nov-24-1999
# /dev/dsk/c0t0d0s3 /dev/rdsk/c0t0d0s3 /master ufs 2 yes -
# /dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0s1 /sybdbs ufs 2 yes -
# /dev/dsk/c0t3d0s4 /dev/rdsk/c0t3d0s4 /syblogs ufs 2 yes -
# /dev/dsk/c0t3d0s5 /dev/rdsk/c0t3d0s5 /sybscurty ufs 2 yes -
# /dev/dsk/c0t0d0s4 /dev/rdsk/c0t0d0s4 /sybtemproc ufs 2 yes -
(7) When I use the format command, I see these two hard drives:
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <IBM-DNES-309170-SA30 cyl 11195 alt 2 hd 5 sec 320>
/iommu#f,e0000000/sbus#f,e0001000/espdma#f,400000/esp#f,800000/sd#0,0
1. c0t3d0 <SEAGATE-ST34520N-1206 cyl 9004 alt 2 hd 4 sec 246>
/iommu#f,e0000000/sbus#f,e0001000/espdma#f,400000/esp#f,800000/sd#3,0
(8) I don’t see any external device attached to the Sybase server. Having said this, there is a backup Sybase server, and the backup Sybase server has an external device attached to it (through a SCSI cable). At this point, I assume the database-files and transaction-files are all stored inside the Sybase server.
By the way, the Sybase server uses this Unix operating system:
SunOS <my-server-name> 5.4 Generic_101945-62 sun4m sparc
And the Sybase version is:
SQL Server/11.0.3.2/P/Sun_svr4/OS 5.4/SWR 7578 Rollup/OPT/Mon Nov 3 22:19:21 PST 1997
By the way, what I have tried so far to repair the database are:
• Tried dbcc checkalloc(, fix). Unfortunately, this command could not fix and could not complete.
• Tried drop-db/add-new-db/restore-db-from-backup. Unfortunately the restore failed to complete.
• Tried fsck-to-fix-the-devices. It could not complete and complained about “MAGIC NUMBER WRONG”.
• Tried Analyze-option-in-format-command-to-repair-Disk-0, and then add-new-db and restore-db-from-backup. This method seemed to work. But after one week or so, I found a table has I/O error. Honestly, I don’t even know if the databases are really in Disk-0 or not.
Please help me to determine which hard drive those database-files and transaction-files are stored, and whether they are in the same hard drive as the Unix operating system.
Thanks in advance.
Jay Chan

What does -2 do when using psftp.exe?

I've just come across this line of code in a .bat file:
psftp -2 -l XXXXX 195.2.37.69 -pw XXXXX -P 10022 -b c:\sftp\sendfile.bat -v -bc -be
The help tells me what all the parameters do except for the -2.
Can anybody tell me what the -2 does?
-2 or -1 forces the use of a the corresponding SSH protocol version, so in your example it enforces the use of SSH2. You can also specify -4 or -6 which forces the use of respectively IPv4 or IPv6.
Quoting the psftp documentation:
3.8.3.16 -1 and -2: specify an SSH protocol version
The -1 and -2 options force PuTTY to use version 1 or version 2 of the
SSH protocol. These options are only meaningful if you are using SSH.
These options are equivalent to selecting your preferred SSH protocol
version as ‘1 only’ or ‘2 only’ in the SSH panel of the PuTTY
configuration box (see section 4.18.4).
So the -2 forces SSH version 2.
In older versions, the psftp tried the SSH version 2 and fell back to the SSH version 1, if the server did not support the version 2. With -2, the fallback to an insecure version 1 did not happen and connection is abandoned. The latest versions do not fall back by default anymore. Nowadays, no serious SSH/SFTP server even supports the version 1 anyway.

Count the number of network packets in realtime (CLI)

I want to count the number of network incoming/outgoing packets in realtime and in the CLI (I want to use the count to write shell script)
I try to use 'tcpdump' and 'tsahrk' combined with pipeline, but it's doesn't work (I have little talent and learning).
Finally,I found the command 'dumpcap', it's almost close to my need, here is dumpcap stdout
[root#hostname ~]# dumpcap -i eth0
File: /tmp/wiresharkXXXXuTE8mU
Packets: 31
When i use this command combined with 'grep', like this 'dumpcap -i eth0 | grep -o '[0-9]*' to filter the number in stdout(31),but it's still doesn't work.
Is their any way to solve this or any idea for this ??
Thank you very much for your attention to this matter.
A packet capture program such as tcpdump, TShark, or dumpcap will probably put the interface into promiscuous mode, which means more packets will show up than would show up under normal circumstances.
The count in dumpcap goes to the standard error, not the standard output.
Dumpcap will write the packets to the file in question, which you may or may not want; if you have a lot of traffic, that file will keep getting bigger and bigger and take up more space on the file system /tmp is on.
You probably want to consider some tool that just gets statistics from the regular networking stack, such as ifconfig on Linux or netstat on various OSes:
$ netstat -w 1 -I en0
input (en0) output
packets errs bytes packets errs bytes colls
0 0 0 0 0 0 0
0 0 0 0 0 0 0
1 0 98 1 0 98 0
1 0 98 1 0 98 0
...
Based on the following output on my computer:
ifconfig eth0 | grep RX
RX packets:113995 errors:0 dropped:0 overruns:0 frame:0
RX bytes:62022831 (62.0 MB) TX bytes:11453699 (11.4 MB)
I can do the following to retrieve the number of RX packets:
ifconfig eth0 | grep "RX packets" | tr ":" " " | awk '{print $3}'
113995
Unfortunately it is not quite clear what exactly it is that you need. What do you mean by "count the number of packets in realtime"? Do you want a counter to be incremented and the value of the counter reported whenever a packet is seen? Or do you simply wish to count how many packets were seen within a given time period? Also what is the ultimate goal of your shell script?

Resources