I want to get the traffic usage of a specified VM over the last month.
I have access to the ESXi and its command prompt and also I have added the ESXi to a VCeter server and have access to it too. Is there any way to get the desired value in a numeric manner in bytes? For example I want to have a number like 100Mb or 1G as the result. Because I want to submit the result to the customer. Any help is appreciated.
You'll probably want Netflow for this. In vSphere it requires the use of a dvSwitch, which I think is limited to certain license levels. You'll also need a collector such as that available from SolarWinds to receive the information so you can see the traffic flows created by the VM.
http://pubs.vmware.com/vsphere-60/topic/com.vmware.vsphere.hostclient.doc/GUID-E19FECAD-8629-4E8A-B61C-1F1C16770B3B.html
http://www.solarwinds.com/netflow-traffic-analyzer
Related
I am looking for a way to programmatically get the name of the vendor that owns a MAC address within a block/range that they purchased. Preferably by querying some API or database, language agnostic. Or if there is some other way that applications do it that I am unaware of.
For example, running nmap -sn 192.168.1.0/24 with root privileges yields
...
Nmap scan report for 192.168.1.111
Host is up (0.35s latency).
MAC Address: B8:27:EB:96:E0:0E (Raspberry Pi Foundation)
...
... and that tells me that the Raspberry Pi Foundation "owns" that MAC Address, within the prefix range that they own: B8:27:EB.
However, I am not sure how nmap knows this, nor how I could find this out myself. Parsing nmap output is not an ideal solution for me. Here's what I found from digging online:
This stackoverflow question references a site that appears to do this, however it appears to not have been updated since 2013, nor does it expose any API endpoints. Most notably, it does not have the newer block of MAC Addresses that the Raspberry Pi Foundation reserved for their newer models (under Raspberry Pi Team, or something along those lines).
I found that the IEEE handles these registrations through their site, however it appears to be for their customers and I could not find an exposed endpoint for their search function.
On that same IEEE page linked above, it looks like I can get a CSV file of their entire database. However that seems large, and would have to be actively kept up-to-date. Does nmap come with an updated database generated from those files locally?
If a public-facing API like I'm envisioning doesn't exist, I'll make one myself for fun. I'd first like to know if I'm thinking about this wrong and if there is an official, "canonical" way that I have not found. Any help would be appreciated, and thank you.
The maintainers of nmap keep a list of prefixes as part of the tool. You can see it here:
https://github.com/nmap/nmap/blob/master/nmap-mac-prefixes
They keep this up to date by periodically importing the public registry on this site:
https://regauth.standards.ieee.org/standards-ra-web/pub/view.html#registries
Note that those files are rate-limited so you should not be querying those csv files ad hoc as part of a software package; rather you should do what nmap does and keep an internal list that you synchronize periodically.
I'm not aware of a publicly available tool to query them as an API; however, creating one that works the same way that nmap does would be fairly trivial. nmap does not update that file more than once or twice a year which makes me suspect that the list doesn't significantly change often enough that keeping your own list would be too onerous (you could even download nmap's list every so often).
First, I'm not a R/RStudio user at all. I'm a Windows admin with the task to configure R and RStudio on a multi-user Citrix environment. To identify users between the multiple sessions, we are using the Palo Alto Terminal Server agent which will allocate a range of ports for each user and use them to identify each users. That's then used to give limited and specific access to resources for each users.
The problem is that the TS Agent also intercept the localhost connection that's created when you start RStudio (process rsession) and RStudio then cannot connect to R. One possible solution to solve this problem is to have control on the ports used when this local session is started.
I have made multiple research on the Internet but I have been unable to find if/how you can change the ports that are used. I have found different config files but none that seem to allow me to fix a single port or a port range.
Any insights on the way to fix the ports for the rsession process so I can better control them? Or another way to look at the problem: do you know the port range used by R/RStudio when they communicate together through the rsession? I can simply avoid using these range with the TS Agent.
I have only skimmed through the RStudio Source code, but it seems that the port is assigned randomly:
https://github.com/rstudio/rstudio/blob/bcc8655ba3676e3155d80296c421362881340a0f/src/node/desktop/src/main/application.ts#L226
However, it also seems like there is a startup parameter --www-port to set the port:
https://github.com/rstudio/rstudio/blob/bcc8655ba3676e3155d80296c421362881340a0f/src/node/desktop/src/main/session-launcher.ts#L592
I am looking for a basic thing yet I have not found not even a single good documentation on getting it done.
I want to allocate a floating IP, then associate it to a network interface of a droplet other than eth0.
The reason is I want to have the ability to very easily switch from one IP to the other with a programming language.
In a few words, I want to be able to do these two commands and both should provide a different response.
curl --interface eth0 https://icanhazip.com
curl --interface eth1 https://icanhazip.com
Also, I want to know what to do once I release the Floating IP, how do I roll back to the starting point.
All documentation I read, rely heavily on "ip route" and "route", most did not even work, some worked but replaced completely the old IP by the floating and that's not what I want, and also they did not show how to rollback the introduced configuration changes.
Please help, I spent 1 whole day now trying to get this to work for a project, and no results so far.
I guess there is no need to know DigitalOcean, how to make this work on other Cloud Providers would apply here too I think.
Update
After asking this on DigitalOcean community forum (https://www.digitalocean.com/community/questions/clear-guide-on-outbound-network-through-floating-ip), they claim that is not supported, although there may be some solutions to this if somebody can provide such a "hacky" solution I would take it too. Thanks
In the cloud (AWS. GCP etc.) ARP is emulated by the virtual network layer, meaning that only IPs assigned to VMs by the cloud platform can be resolved. Most of the L2 failover protocols do break for that reason. Even if ARP worked,the IP allocation process for these IPs (often called “floating IPs”) would not integrate with the virtual network in a standard way, so your OS can't just "grab" the IP using ARP and route the packets to itself.
I have not personally done this on Digital Ocean, but I assume that you can call the cloud's proprietary API to do this functionality if you would like to go this route.
See this link on GCP about floating IPs and their implementation. Hope this is helpful.
Here's an idea that needs to be tested:
Let's say you have Node1(10.1.1.1/24) and Node2(10.1.1.2/24)
Create a loopback interface on both VMs and set the same IP address for both like (10.2.1.1/32)
Start a heartbeat send/receive between them
When NodeA starts it automatically makes an API call to create a route for 10.2.1.1/32 and points to itself with preference 2
When NodeB starts it automatically makes an API call to create a route for 10.2.1.1/32 and points to itself with preference 1
The nodes could monitor each other to withdraw the static routes if the other fails. Ideally you would need a 3rd node to reach quorum and prevent split brain scenarios, but you get the idea right?
I have big access log (~30Gb) and I'm looking for ways for find suspicious/bot IP addresses. Of course, we can replace IP with (IP + User_Agent). So my questions are:
find average requests number that are done from any IP
find IP addresses that are making more requests than average (see previous point)
find IP addresses that are doing requests regulary (every hour for example) during day
you recommendations about how to detect bot
This log is rather big and I don't think that R lang could process it. Should I use some kind of storage behind R (hadoop or something similar)? I absolutely have no experience in processing/analyzing big data so any ideas, recommedations & tuts/articles are appreciated.
The access log probably contains a lot of data which you may not need based on your question, if you only care about the time of the request and the orginating IP you could easily reduce the data size by extracting "columns" from the input before reading it into R, some standard command line tools such as cut or awk should do the trick.
If you want to keep more details another option could be to load the access log into a database and use this for further processing, 30GB is not a lot for a database, but of course this means some additional work: design a datbase schema and a way to load the data in the database.
You can also do the following type of analysis
Getting the geo location of IP addresses and comparing access frequency based on geo_location + time at geo_location (the access frequency could be normal during day time at the geo location but not after midnight)
If you have username information, check whether multiple IP addresses are using the same username during same time period
WSO2 has done some Anomaly Detection work using their Analytics Platform which is pretty scalable for most anomaly detection scenarios. Check it out - http://wso2.com/analytics/solutions/fraud-and-anomaly-detection-solution/
This might be a better option than doing through R, since it allows you to do complex event processing (through SQL like queries) as well as machine learning.
You can also do the following type of analysis
a) If the IP address is from data center range, it is likely from a bot than normal user.
b) If the IP address is from search engine range, it is high likely from a search engine bot.
You can get the geolocation database from IP2Location which has the usage type information to detect data center or search engine.
Check goaccess.io - works for me. With logs for different websites, distributed on several servers. Allows usage of GEOiP and identifies bots out of the box.
Check out https://ipdetective.io it tracks IP addresses that originate from datacenters, vpns, proxies, tor node and bot nets. It offers a free API as well so you can test it out.
I am trying to find the proper way of accomplishing the following.
I would like to provide 2Gb/s access for clients accessing a fileserver guest vm on a ESXi server, which itself access the datastore over iSCSI. Therefore the ESXi server need 2Gbps connection to the NAS. I would also like to provide 2Gbps directly on the NAS.
Looks like there are three technology which can help. Link aggregation (802.3ad, LAG, Trunk), Multi Path IO (MPIO), and iSCSI Multiple connection per session (MC/S).
However each have their own purpose and drawbacks, Aggregation provide 2Gbps total but a single connection (I think it's based on source/dest MAC address) can only get 1Gbps, which is useless (I think for iSCSI for example which is a single stream), MPIO seem a good option for iSCSI as it balance any traffic on two connection however it seem to require 2 IPs on the Source and 2 IPs on the DEST, I am unsure about MCs.
Here is what I would like to archive, however I am not sure of the technology to employ on each NIC pair of 1Gbps.
I also think this design is flawed because doing link aggregation between the NAS and the switch would prevent me from using MPIO on the ESX as it also require 2 IP on the nas and I think link aggregation will give me a single IP.
Maybe using MCs instead of MPIO would work?
Here a diagram:
If you want to achieve 2Gbps to a VM in ESX it is possible using MPIO & iSCSI but as you say you will need two adapters on the ESX host and two on the NAS. The drawback is that your NAS will need to support multiple connections from the same initiator, not all of them do. The path policy will need to be set to round-robin so you can use Active-Active connections. In order to get ESX to use both paths # over 50% each you will need to adjust the round robin balancing mode to switch paths every 1 IOPS instead of 1000. You can do this by SSHing to the host and using esxcli (if you need full instructions on how to do that I can provide them).
After this you should be able to run IOMeter on a VM and see the data rate # over 1Gbps, maybe 150MB/s for 1500 MTU and if you are using jumbo frames, then you will get around 200MB/s.
On another note (which might prove useful to your setups in the future), it is possible to achieve 2Gbps with two adapters on the source and bonded adapter on the NAS (so 2 → 1) when using the MPIO iSCSI Initiator that comes with Server 2008. This initiator works slightly different to VMWare and doesn't require your NAS to support many connections from one initiator — from what I can tell it spawns multiple initiators instead of sessions.