I am using multiple conf files inside my main conf file. However, the service is constantly taking on the first conf file.
Below is my main conf file code,
#INCLUDE fluent-bit-for-application-info-log.conf
#INCLUDE fluent-bit-for-application.conf
I have tested with different orders but every time it is constantly picking up only the first line.
[SERVICE]
flush 1
log_level debug
parsers_file parsers.conf
plugins_file plugins.conf
storage.metrics on
[INPUT]
name tail
path D:\xxxx\**\*.log
parser iis
Multiline Off
Skip_Empty_Lines On
refresh_interval 10
Tag iis_log
[OUTPUT]
Name s3
Match iis_log
Bucket xxxx
region ap-xx-1
s3_key_format /xxxx/xxxxx/%Y/%m/%d/%H/$UUID.log
Is there any additional config we need to add to process all config files?
Note: All my config files have the same code except path, bucket and tag details.
Please let me know if you need any information.
Thanks in Advance.
I have installed Mod Security 3 on my Nginx server.
Up to date and seeing multiple pages on how to put an IP on the white list, I find that all docs talk to me about doing something, and that causes me a configuration error in nginx.
whitelist.conf (call on modsecurity.conf after read all rules files)
SecRule REMOTE_ADDR "#ipMatch 192.168.110" phase:1,nolog,allow,ctl:ruleEngine=Off
Error
root#hq:/etc/nginx/modsec/whitelist# nginx -t
nginx: [emerg] "modsecurity_rules_file" directive Rules error. File: /etc/nginx/modsec/whitelist/whitelist.conf. Line: 1. Column: 19. in /etc/nginx/conf.d/default.conf:4
Some body can help me?
main.conf
# From https://github.com/SpiderLabs/ModSecurity/blob/master/
# modsecurity.conf-recommended
#
# Edit to set SecRuleEngine On
Include "/etc/nginx/modsec/modsecurity.conf"
# Basic test rule
# SecRule ARGS:testparam "#contains test" "id:1234,deny,status:403"
# OWASP CRS v3 rules
Include /usr/local/owasp-modsecurity-crs-3.0.2/crs-setup.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-901-INITIALIZATION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-905-COMMON-EXCEPTIONS.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-910-IP-REPUTATION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-911-METHOD-ENFORCEMENT.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-912-DOS-PROTECTION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-913-SCANNER-DETECTION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-921-PROTOCOL-ATTACK.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-949-BLOCKING-EVALUATION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-950-DATA-LEAKAGES.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-959-BLOCKING-EVALUATION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-980-CORRELATION.conf
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
# Mias
Include /usr/local/owasp-modsecurity-crs-3.0.2/rules/REQUEST-903.9002-WORDPRESS-EXCLUSION-RULES.conf
# Whitelisting
Include /etc/nginx/modsec/whitelist/*.conf
Within CRS 3.x two files are provided to help you add these different rule modifications, they are: rules/REQUEST-00-LOCAL-WHITELIST.conf.example and rules/RESPONSE-99-EXCEPTIONS.conf.example. As is noted in the Installing OWASP CRS documentation, the .example extension is provided specifically so that when these files are renamed, future updates will not overwrite these files. As is listed within the Installing OWASP CRS documentation, before adding a whitelist or exception modification you should rename these files to end in the .conf exception.
https://www.modsecurity.org/CRS/Documentation/exceptions.html
We're got a pre-existing rsyslog config file which is working for papertrail e.g.
/etc/rsyslog.d/20-papertrail.conf which has
*.* #logs4.papertrailapp.com:44407
However we've got a couple of NGINX websites on the server so would like to have it also monitor their error logs.
The paths to them are:
/var/log/nginx/www.website-one.com-error.log
/var/log/nginx/www.website-two.com-error.log
/var/log/nginx/www.website-three.com-error.log
However this /var/log/nginx also contains a bunch of .log files which we do not want to monitor e.g.
/var/log/nginx/error.log
/var/log/nginx/access.log
/var/log/nginx/error.log1
/var/log/nginx/nginx.log
In my head we need to add something like...
/var/log/nginx/*-error.log
And make sure they pipe to the papertrail url as well.
However I'm struggling to decipher the rsyslog documentation to figure out how to do this.
Thanks!
In rsyslog documentation it seems that you can use wildcards in files.
File
The file being monitored. So far, this must be an absolute name (no macros or templates). Note that wildcards are supported at the file name level (see WildCards below for more details).
WildCards
Before Version: 8.25.0
Wildcards are only supported in the filename part, not in directory names.
/var/log/*.log works.
/var/log/*/syslog.log does not work.
Since Version: 8.25.0
Wildcards are supported in filename and paths which means these samples will work:
/var/log/*.log works.
/var/log/*/syslog.log works.
/var/log/*/*.log works.
All matching files in all matching subfolders will work. Note that this may decrease performance in imfile depending on how many directories and files are being watched dynamically.
If you want to forward your vhosts logs you can change configuration directly in NGINX vhosts configuration, you should change/add access_log and error_log policies as explained here or use custom facilities to forward your logs (using rsyslog).
HOW TO DO IT USING RSYSLOG?
Create a new custom file in /etc/rsyslog.d/nginx_custom.conf:
module(load="imfile" PollingInterval="1") #needs to be done just once
# File 1
input(type="imfile"
File="/var/log/nginx/www.website-*.com-error.log"
Tag="websites"
Facility="local0")
local0.* #logs4.papertrailapp.com:44407
#Just to test that logs are forwarded, comment the line once you've tested it
local0.* /var/log/test.log
And restart rsyslog service
NOTE: Line local0.* /var/log/test.log is just to test that you can see forwarded logs into your local server, comment this line after you've tested that everything works.
I am installing openldap from scratch on a Centos7
I followed the quick start guide on the openldap.org website http://www.openldap.org/doc/admin24/quickstart.html
Everything worked fine up until step 7, I installed openldap in a customized directory without problem
below are the command I ran
cd /usr/local/src
mkdir openldap
cd openldap
cp /home/user/openldap/src/* . //copied everything in the tar ball to the directory
yum install gcc openssl libdb-devel libtool-ltdl-devel.x86_64 openssl-devel.x86_64 cyrus-sasl-devel.x86_64
//yum installed every required package, including devel package for make
./configure --prefix=/usr/local/src/openldap-install //configure to install in custom location
make depend
make
make test //all successful
make install
Then I tried to follow step 8, editing the slapd.ldif to put in my own domain, dc=adomain,dc=com , in my case, was in /usr/local/src/openldap-install/etc/openldap/slapd.ldif
I then added the /usr/local/src/openldap-install/sbin/ directory to the PATH environment, by inserting a openldap_path.sh in the profile.d dir, verified that it worked.
then I followed step 9, (I was running everything by root)
slapadd -F /usr/local/etc/cn=config -l /usr/local/etc/openldap/slapd.ldif
but it returned "available database(s) do not allow slapadd new server"
I don't understand, I am not migrating (as suggested in some other stackoverflow post), its a brand new server, but it just doesn't allow me to continue.
Any help would be appreciated, I just don't know what to do next.
#EJP
File as follow
#
# See slapd-config(5) for details on configuration options.
# This file should NOT be world readable.
#
dn: cn=config
objectClass: olcGlobal
cn: config
#
#
# Define global ACLs to disable default read access.
#
olcArgsFile: /usr/local/src/openldap-install/var/run/slapd.args
olcPidFile: /usr/local/src/openldap-install/var/run/slapd.pid
#
# Do not enable referrals until AFTER you have a working directory
# service AND an understanding of referrals.
#olcReferral: ldap://root.openldap.org
#
# Sample security restrictions
# Require integrity protection (prevent hijacking)
# Require 112-bit (3DES or better) encryption for updates
# Require 64-bit encryption for simple bind
#olcSecurity: ssf=1 update_ssf=112 simple_bind=64
#
# Load dynamic backend modules:
#
#dn: cn=module,cn=config
#objectClass: olcModuleList
#cn: module
#olcModulepath: /usr/local/src/openldap-install/libexec/openldap
#olcModuleload: back_bdb.la
#olcModuleload: back_hdb.la
#olcModuleload: back_ldap.la
#olcModuleload: back_passwd.la
#olcModuleload: back_shell.la
dn: cn=schema,cn=config
objectClass: olcSchemaConfig
cn: schema
include: file:///usr/local/src/openldap-install/etc/openldap/schema/core.ldif
# Frontend settings
#
dn: olcDatabase=frontend,cn=config
objectClass: olcDatabaseConfig
objectClass: olcFrontendConfig
olcDatabase: frontend
#
# Sample global access control policy:
# Root DSE: allow anyone to read it
# Subschema (sub)entry DSE: allow anyone to read it
# Other DSEs:
# Allow self write access
# Allow authenticated users read access
# Allow anonymous users to authenticate
#
#olcAccess: to dn.base="" by * read
#olcAccess: to dn.base="cn=Subschema" by * read
#olcAccess: to *
# by self write
# by users read
# by anonymous auth
#
# if no access controls are present, the default policy
# allows anyone and everyone to read anything but restricts
# updates to rootdn. (e.g., "access to * by * read")
#
# rootdn can always read and write EVERYTHING!
#
#######################################################################
# LMDB database definitions
#######################################################################
#
dn: olcDatabase=mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: mdb
**olcSuffix: dc=ldap,dc=domain,dc=com**
**olcRootDN: cn=Manager,dc=ldap,dc=domain,dc=com**
# Cleartext passwords, especially for the rootdn, should
# be avoided. See slappasswd(8) and slapd-config(5) for details.
# Use of strong authentication encouraged.
**olcRootPW: {SSHA}strong**
# The database directory MUST exist prior to running slapd AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
**olcDbDirectory: /usr/local/src/openldap-install/var/openldap-data**
# Indices to maintain
olcDbIndex: objectClass eq
only the last few lines were changed from the slapd.ldif.default file
and below is the file tree in the installation path output by ls -R
.:
bin etc include lib libexec sbin share var
./bin:
ldapadd ldapdelete ldapmodify ldappasswd ldapurl
ldapcompare ldapexop ldapmodrdn ldapsearch ldapwhoami
./etc:
openldap
./etc/openldap:
ldap.conf schema slapd.conf.default slapd.ldif.default
ldap.conf.default slapd.conf slapd.ldif
./etc/openldap/schema:
collective.ldif cosine.schema java.ldif openldap.schema
collective.schema duaconf.ldif java.schema pmi.ldif
corba.ldif duaconf.schema misc.ldif pmi.schema
corba.schema dyngroup.ldif misc.schema ppolicy.ldif
core.ldif dyngroup.schema nis.ldif ppolicy.schema
core.schema inetorgperson.ldif nis.schema README
cosine.ldif inetorgperson.schema openldap.ldif
./include:
lber.h ldap_cdefs.h ldap.h ldap_utf8.h slapi-plugin.h
lber_types.h ldap_features.h ldap_schema.h ldif.h
./lib:
liblber-2.4.so.2 libldap-2.4.so.2 libldap_r-2.4.so.2.10.7
liblber-2.4.so.2.10.7 libldap-2.4.so.2.10.7 libldap_r.a
liblber.a libldap.a libldap_r.la
liblber.la libldap.la libldap_r.so
liblber.so libldap_r-2.4.so.2 libldap.so
./libexec:
slapd
./sbin:
slapacl slapauth slapdn slappasswd slaptest
slapadd slapcat slapindex slapschema
./share:
man
./share/man:
man1 man3 man5 man8
./share/man/man1:
ldapadd.1 ldapdelete.1 ldapmodify.1 ldappasswd.1 ldapurl.1
ldapcompare.1 ldapexop.1 ldapmodrdn.1 ldapsearch.1 ldapwhoami.1
./share/man/man3:
ber_alloc_t.3 ldap_free_urldesc.3
blablablablabla
blablablablabla
blablablablabla
./share/man/man5:
ldap.conf.5 slapd-mdb.5 slapd-sql.5 slapo-ppolicy.5
blablablablabla
blablablablabla
blablablablabla
./share/man/man8:
blablablablabla
blablablablabla
./var:
run
./var/run:
EDIT
#EJP I sort of get what you mean. you are saying the default LDIF file is actually the cn=config database?
I am nearly giving up, 1 week of my time and not getting an ldap server starting. I don't see what is the problem, Perhaps cleaning up some concept of mine can actually solve the situation here?
How to define my own database? I thought I had already done that by changing the olcsuffix?
how is the cn=config database different from a "configuration database"?
what file do I edit / create to define my own database?
What does it mean in step 8 of the openldap quickstart guide contain a MDB database definition of the form ?
What does it mean in step 9 of the openldap quickstart guide import your configuration database? does it mean the installed openldap server had no configuration?
What does the slapd.conf file that came along with the installation do? does it contain "definition" or "configuration"?
I have tried to skip step 8 and 9 and directly tried to start SLAPD, but it always failed, should SLAPD be able to start on its own without me "importing my own configuration database?
Thankyou EJP for your great patience with me, I know I am missing something extremely trivial, but believe me I have dug through the internet for days (and book store for one day) failing to find definitive answer. Would be really great if you can clarify with me some concepts by answer my questions or perhaps your own explanation, or provide me with just a dummy ldif file that I can do slapadd then I can figure out whats wrong.
I attribute my difficulty to I reckon poorly done openldap guidance documentation:
They have stated that slapd.d is the new way to go, but the compiled source make does not generate the slapd.d/ directory, nor did I find options in the ./configure to switch to such
bdb and hdb seems fading out and replaced by mdb, but ./configure defaults enabled those two, requiring some extra libdb-devel package to correctly make them (I am using Centos 7).
the sub directory required by example setup files slapd.conf slapd.ldif(and until today I don't know if they are example files or working files) is not setup by make install, even /var/openldap-data
not even sub directory required by the quickstart guide command was created by make install script
I am not sure if that should be the right, but I believe SLAPD should be shipped at least "working out of the box" and subsequent configuration should be issued by ldapmodify / add / del command.
I didn't figure out why the slapadd command failed when I hit the same error. But,if what you want is to get your LDAP server works, I can share what I did.
Step 1: Configure your MDB in slapd.conf file directly.
When OpenLDAP is installed, a slapd.conf file under /usr/local/etc/openldap has already been created and configured with a MDB database. You can change the MDB setting according to your need. And this is the default setting :
#######################################################################
# MDB database definitions
#######################################################################
database mdb
maxsize 1073741824
suffix "dc=my-domain,dc=com"
rootdn "cn=Manager,dc=my-domain,dc=com"
# Cleartext passwords, especially for the rootdn, should
# be avoid. See slappasswd(8) and slapd.conf(5) for details.
# Use of strong authentication encouraged.
rootpw secret
# The database directory MUST exist prior to running slapd AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory /usr/local/var/openldap-data
# Indices to maintain
index objectClass eq
Step 2: Start up slapd after configuration.
Command to start slapd:
/usr/local/libexec/slapd -d 1
If you see "slapd starting", it means slapd is started.
Step 3: Create DB entries.
You can refer to the process given in OpenLDAP Administrator guide :
1.Create an entries.ldif file like this:
# Organization for Example Corporation
dn: dc=example,dc=com
objectClass: dcObject
objectClass: organization
dc: example
o: Example Corporation
description: The Example Corporation
# Organizational Role for Directory Manager
dn: cn=Manager,dc=example,dc=com
objectClass: organizationalRole
cn: Manager
description: Directory Manager
2.Create the entries with ldapadd tool:
ldapadd -f entries.ldif -x -D "cn=Manager,dc=example,dc=com" -w secret
Once you've done the upper 3 steps, the LDAP server is ready for use. You can add DIT for your use.
FYI, a good GUI client can help you a lot. I used Jarek Gawor’s excellent LDAP Browser/Editor v2.8.2
Hope this answer can help you or others in future.
Turns out this answer solves the problem Openldap naming context issue with Apache directory studio?.
I think I appreciate that people can answer some specifics of my question or provide an example for answer. Not all people learn by worded explanations, some learn by examples and reverse engineering. Read manuals and only ask question when you don't understand, I would think that is such a prior statement, and most have gone past that stage before actually asking any questions.
Yes, I think I should ask on serverfault, since stackoverflow is more geared towards programming.
I just ran into this using OpenLDAP 2.6.0 ; following "configure; make; make install" I wasn't able to run slapd, it failed server startup with "unknown file on line 72"
Turns out it didn't create a /usr/local/var/openldap-data/ directory.
mkdir /usr/local/var/openldap-data/ resolved it.
You should convert your default slapd.conf into new slapd.d directory format. Add config database definition in slapd.conf:
# before the first database definition
database config
# NOTE: the suffix is hardcoded as cn=config and
# MUST not have a suffix directive
# normal rules apply - rootdn can be anything you want
# but MUST be under cn=config
rootdn "cn=admin,cn=config"
# use any of the supported password formats e.g. {SSHA} etc
# or plaintext as shown
rootpw config
and then convert it using
slaptest -f /usr/local/etc/openldap/slapd.conf -F /usr/local/etc/openldap/slapd.d
and then delete slapd.conf. Then you can continue your configuration with slapadd. Take a look at Configuration Guide or http://www.zytrax.com/books/ldap/ch6/slapd-config.html.
Hope this will help you.
Following the documentation noted in the wiki, I'm trying to use the KeyczarTool to generate new keyset. Anyone else come across this FileNotFoundException? The KeyczarTool.jar has rwx permissions and tried running via sudo.
From docs
Command Usage:
create --location=/path/to/keys --purpose=(crypt|sign) [--name="A name"] [--asymmetric=(dsa|rsa|ec)]
Creates a new, empty key set in the given location.
This key set must have a purpose of either "crypt" or "sign"
and may optionally be given a name. The optional version
flag will generate a public key set of the given algorithm.
The "dsa" and "ec" asymmetric values are valid only for sets
with "sign" purpose.
Cmd:
$ java -jar KeyczarTool-0.71f-060112.jar create --location=/keys --purpose=crypt -name="first key" --asymmetric=rsa
output:
org.keyczar.exceptions.KeyczarException: Unable to write to: /keys/meta
at org.keyczar.KeyczarTool.create(KeyczarTool.java:366)
at org.keyczar.KeyczarTool.main(KeyczarTool.java:123)
Caused by: java.io.FileNotFoundException: /keys/meta (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:145)
at org.keyczar.KeyczarTool.create(KeyczarTool.java:362)
... 1 more
With the current version of java keyczar the directory "keys" needs to be created first before running the program.
This is a known issue KeyczarTool should create directories automatically.
As #jbtule kindly pointed out you must create the keys dir first. But also include . before the slash.
Correct working command is:
$ java -jar KeyczarTool-0.71f-060112.jar create --location=./keys --purpose=crypt -name="first key" --asymmetric=rsa