Using squid also to cache "big" files (for big I mean >20Mb), happens that if a pc download a file, first time that file has downloaded from internet. Second time from squid cache, but: if I try to download same file from another pc, same file come downloaded from internet and not via squid.
maximum_object_size 5 Gb
cache_dir ufs /data/vmware/squid-cache 30720 16 256
cache_mem 4096 MB
minimum_object_size 0
maximum_object_size_in_memory 512 Kb
cache_replacement_policy heap GDSF
cache_swap_low 85
cache_swap_high 90
half_closed_clients off
hosts_file /etc/hosts
memory_pools off
client_db off
dns_nameservers 127.0.0.1
via off
forwarded_for off
httpd_suppress_version_string off
follow_x_forwarded_for deny all
#visible_hostname sign.bunker.org
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private
refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-no-store ignore-private
refresh_pattern -i \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private
refresh_pattern -i \.index.(html|htm)$ 0 40% 10080
refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320
refresh_pattern . 0 40% 40320
refresh_pattern -i movies.com/.* 10080 90% 43200
refresh_pattern (/cgi-bin/|\?) 0 0% 0
Related
What is the meaning of the output under the NAME tag of the lsof -i
command when it appears as such *:*
Example output:
The UDP *:* means you have processes on your system listening on all network interfaces for IPv4 UDP packets:
All IPv4 Addresses are represented by a * on the left of the colon
All ports are represented by a * on the right of the colon
You can test this by sending a batch of 5 UDP packets to one of your local addresses with netcat:
$ echo "data" | nc -w1 -u 127.0.0.1 1-5
While using lsof in repeat mode to show the UDP connection activity:
$ lsof -r1 -iUDP -P -n | grep -E "^nc"
nc 3197 root 3u IPv4 0x614d3be71aa32a89 0t0 UDP 127.0.0.1:57137->127.0.0.1:2
nc 3197 root 3u IPv4 0x614d3be71b503dc9 0t0 UDP 127.0.0.1:62455->127.0.0.1:3
nc 3197 root 3u IPv4 0x614d3be71bdbe351 0t0 UDP 127.0.0.1:52982->127.0.0.1:4
nc 3197 root 3u IPv4 0x614d3be71b6335b9 0t0 UDP 127.0.0.1:61450->127.0.0.1:5
I am trying to build transparent proxy with squid on OpenBSD 5.7 with pf firewall.
I compile squid from source with below options:
$ squid -v
Squid Cache: Version **3.4.13**
configure options:
--prefix=/usr/local/squid
--with-default-user=squid
--enable-icmp
--enable-storeio=ufs,aufs
--enable-removal-policies=lru,heap
--disable-snmp
--disable-wccp
--disable-wccpv2
--enable-pf-transparent
--enable-ipv6
--enable-referer-log
--with-nat-devpf
--enable-debug-cbdata
--enable-useragent-log
--enable-refererlog
--enable-cache-digests
--with-large-files
--with-pthreads
--without-mit-krb5
--without-heimdal-krb5
--without-gnugss
--disable-eui
--disable-auth
--enable-ltdl-convenience
$ uname -a
OpenBSD dns.localdomain 5.7 GENERIC#825 amd64
My squid.conf:
visible_hostname dns.local
acl localnet src 192.168.1.0/24 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
# allow
http_access allow localnet
http_access allow localhost
# And finally deny all other access to this proxy
http_access deny all
# Squid normally listens to port 3128
http_port 3128
http_port 127.0.0.1:3129 intercept
# disk cache directory.
cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256
# Leave coredumps in the first cache dir
coredump_dir /usr/local/squid/var/cache/squid
#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
Enabled gateway to connect Internet:
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
pf.conf:
int_if = "vic1"
ext_if = "vic0"
lan_net = "192.168.1.0/24"
# Settings
set block-policy return
set loginterface egress
set skip on lo
# NAT
match out on egress inet from !(egress:network) to any nat-to (egress:0)
pass in quick log on $ext_if inet proto tcp from 192.168.1.0/24 to port www divert-to 127.0.0.1 port 3129
pass out quick log inet from 192.168.1.0/24 divert-reply
#
# Rules
#
block all
# allow dns
pass quick on {$int_if, $ext_if} inet proto udp from {self, $lan_net} to any port 53
# allow local access to web
pass quick on $ext_if inet proto tcp from {self} to any port 80
# allow icmp
pass quick on $int_if inet proto icmp from $lan_net to any
# allow ssh from $ext_if
pass quick on $ext_if inet proto tcp from any to ($ext_if) port 22
I think problem in pf rule. Because pf can not divert packet to port 3129? I've tested with command:
nc -l 3129
but it didn't response any HTTP header.
The rule in Squid wiki can not apply to pf because syntax error.
Thank You in advance
Maybe you mean $int_if instead of $ext_if on this rule? :
pass in quick log on $ext_if inet proto tcp from 192.168.1.0/24 to port www divert-to 127.0.0.1 port 3129
As I understand, you want to divert traffic comming from internal network to local port 3129.
I'm trying to setup Kubernetes in Openstack + CoreOS.
I have master 10.240.63.84 and 2 minions .63 and .83. I also created 3 redis pods:
redis-gopher-gziey 10.244.32.2 10.240.63.66/10.240.63.66
redis-managed-oh43e 10.244.32.3 10.240.63.66/10.240.63.66
redis-primary-fplln 10.244.54.2 10.240.63.83/10.240.63.83
master's routing table looks like:
10.240.63.0 * 255.255.255.0 U 0 0 0 eth0
10.240.63.1 * 255.255.255.255 UH 1024 0 0 eth0
10.244.0.0 * 255.255.0.0 U 0 0 0 flannel.1
10.244.50.0 * 255.255.255.0 U 0 0 0 docker0
and output of ifconfig -a is :
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 10.244.50.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::542f:6fff:fe4a:adf3 prefixlen 64 scopeid 0x20<link>
ether 56:84:7a:fe:97:99 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 90 (90.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.240.63.84 netmask 255.255.255.0 broadcast 10.240.63.255
inet6 fe80::f816:3eff:fe89:e9a0 prefixlen 64 scopeid 0x20<link>
ether fa:16:3e:89:e9:a0 txqueuelen 1000 (Ethernet)
RX packets 430706 bytes 559764129 (533.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 238519 bytes 116083693 (110.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.50.0 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::601f:62ff:feed:1556 prefixlen 64 scopeid 0x20<link>
ether 62:1f:62:ed:15:56 txqueuelen 0 (Ethernet)
RX packets 20 bytes 1504 (1.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 79 bytes 7686 (7.5 KiB)
TX errors 0 dropped 19 overruns 0 carrier 0 collisions 0
Flanneld config used for initialization is:
Master:
- name: flanneld.service
command: start
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
ExecStart=
ExecStart=/usr/libexec/sdnotify-proxy /run/flannel/sd.sock \
/usr/bin/docker run --net=host --privileged=true --rm \
--volume=/run/flannel:/run/flannel \
--env=NOTIFY_SOCKET=/run/flannel/sd.sock \
--env-file=/run/flannel/options.env \
--volume=${ETCD_SSL_DIR}:/etc/ssl/etcd:ro \
quay.io/coreos/flannel:${FLANNEL_VER} /opt/bin/flanneld --ip-masq=true --iface=eth0
Minion:
- name: flanneld.service
command: start
drop-ins:
- name: 50-network-config.conf
content: |
[Service]
ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}'
ExecStart=
ExecStart=/usr/libexec/sdnotify-proxy /run/flannel/sd.sock \
/usr/bin/docker run --net=host --privileged=true --rm \
--volume=/run/flannel:/run/flannel \
--env=NOTIFY_SOCKET=/run/flannel/sd.sock \
--env-file=/run/flannel/options.env \
--volume=${ETCD_SSL_DIR}:/etc/ssl/etcd:ro \
quay.io/coreos/flannel:${FLANNEL_VER} /opt/bin/flanneld -etcd-endpoints http://10.240.63.84:4001 --ip-masq=true --iface=eth0
So the issue is that i can't ping any of the pods from master, as well as connect to any port, error is:
ncat -v -t 10.244.32.2 6379
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: No route to host.
This sort of thing is hard to debug remotely. Things I would check:
1) on the sender: iptables -t raw -I OUTPUT -d 10.244.32.2 -j TRACE; dmesg -c > /dev/null; ncat -v -t 10.244.32.2 6379; dmesg;
This will give you some insight into what the kernel is doing.
2) on the sender: tcpdump -i any host 10.244.32.2 & ncat -v -t 10.244.32.2 6379;`
This will give a bit more insight.
3) on the receiver: iptables -t raw -I OUTPUT -d 10.244.32.2 -j TRACE; dmesg -c > /dev/null; ncat -v -t 10.244.32.2 6379; dmesg;
This will tell you if the packet came through the encapsulation.
You need to basically prove the plumbing through the whole connection.
Is there any way on Linux to get detailed analysis of network traffic to and from a process . Detailed analysis means the data as provided by tcpdump, wireshark.
How about using lsof to get further information on the process?
lsof -c $process
to see what files and network connections that are used by the process.
lsof -i | egrep 'mongod.*LISTEN'
to grep for all connections to the mongod process. From this you can see what port the process is listening on
$ lsof -i | egrep 'mongod.*LISTEN'
mongod 64223 fred 6u IPv4 0xffffff80298fec20 0t0 TCP *:27017 (LISTEN)
mongod 64223 fred 7u IPv4 0xffffff8017393500 0t0 TCP *:28017 (LISTEN)
mongod 64224 fred 9u IPv4 0xffffff8014038500 0t0 TCP *:27018 (LISTEN)
mongod 64224 fred 10u IPv4 0xffffff801746f880 0t0 TCP *:28018 (LISTEN)
......
......
......
and so then run tcpdump on that port.
HTH!
I am getting a "TCP connection to localhost:8090 failed: Connection refused" error when trying to use ffserver on Ubuntu 10.04LTS Desktop.
I am typing:
ffmpeg -f video4linux2 -s 640x480 -r 30000/1001 -i /dev/video0 -f avi -vcodec mjpeg -r 30000/1001 http://localhost:8090/feed1.ffm
Here is the full output:
Input #0, video4linux2, from '/dev/video0':
Duration: N/A, start: 1314207657.841770, bitrate: N/A
Stream #0.0: Video: mjpeg, yuvj420p, 1280x720 [SAR 96:96 DAR 16:9], -5 kb/s, 30 tbr, 1000k tbn, 30 tbc
[tcp # 0x9e58980] TCP connection to localhost:8090 failed: Connection refused
[buffer # 0x9e58260] w:1280 h:720 pixfmt:yuvj420p tb:1/1000000 sar:96/96 sws_param:
Output #0, avi, to 'http://localhost:8090/feed1.ffm':
Metadata:
ISFT : Lavf53.8.0
Stream #0.0: Video: mjpeg, yuvj420p, 1280x720 [SAR 96:96 DAR 16:9], q=2-31, 200 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
Stream #0.0 -> #0.0`
ffserver seems to acknowledge the request though:
Wed Aug 24 13:40:57 2011 127.0.0.1 - - [POST] "/feed1.ffm HTTP/1.1" 200 1356
Where is the problem? How can I use ffmpeg correctly?
I was having the same issue, here's what worked for me:
in the /etc/ffserver.conf file put:
BindAddress 0.0.0.0
then in the <feed></feed> section add:
ACL allow 127.0.0.1
ACL allow localhost
ACL allow 192.168.0.0 192.168.255.255
assuming your network ip is 192.168.x.x
then start your server:
ffserver -d -f /etc/ffserver.conf
Sounds like you are not allowed connect or possibly that the service is not there.
You sure you have
Started ffserver and that it is on 8090?
That you have set
ACL allow 127.0.0.1
or
ACL allow localhost
in you ffserver configuration file in the clause?