Database logging with PingFederate - pingfederate

Question

I have changed the logging in PingFederate to log to the database. Not only have I done this for the four log files that have the configuration prepared (audit, provisioner, provisioner_audit & server) I have also added appenders for all other log files with the exception of init.log. They are all logging to the database but there is missing data in most of the logs. My root section looks like the following:
<Root level="INFO" includeLocation="false">
<AppenderRef ref="ServerLogToSQLServerDB-FAILOVER" />
<AppenderRef ref="ProvisionerLogToSQLServerDB-FAILOVER" />
<AppenderRef ref="OutboundProvisionerEventToSQLServerDB-FAILOVER" />
<AppenderRef ref="SecurityAuditToSQLServerDB-FAILOVER" />
<AppenderRef ref="AdminLogToSQLServerDB-FAILOVER" />
<AppenderRef ref="AdminApiLogToSQLServerDB-FAILOVER" />
<AppenderRef ref="RuntimeApiLogToSQLServerDB-FAILOVER" />
<AppenderRef ref="TransactionLogToSQLServerDB-FAILOVER" />
<AppenderRef ref="AdminEventDetailLogToSQLServerDB-FAILOVER" />
</Root>
and my log4j2.db.properties file is also populated (obviously or I wouldn't have anything). I have also turned on the admin-event-detail log and that is logging too. I will use the audit log as an example and the JDBC Appender looks like this:
<JDBC name="SecurityAuditToSQLServerDB" tableName="log_audit" ignoreExceptions="false">
<ConnectionFactory class="com.pingidentity.log4j.LoggingDBConnectionPoolFactory" method="getSQLServerDatabaseConnection" />
<Column name="dtime" isEventTimestamp="true" />
<Column name="event" pattern="%X{event}" />
<Column name="username" pattern="%X{subject}" />
<Column name="ip" pattern="%X{ip}" />
<Column name="app" pattern="%X{app}" />
<Column name="host" pattern="%X{host}" />
<Column name="protocol" pattern="%X{protocol}" />
<Column name="role" pattern="%X{role}" />
<Column name="partnerid" pattern="%X{connectionid}" />
<Column name="status" pattern="%X{status}" />
<Column name="adapterid" pattern="%X{adapterid}" />
<Column name="description" pattern="%X{description}" />
<Column name="responsetime" pattern="%X{responsetime}" />
<Column name="trackingid" pattern="%X{trackingid}" />
</JDBC>
<RollingFile name="SecurityAuditToSQLServerDB-FILE" fileName="${sys:pf.log.dir}/audit-sqlserver-failover.log"
filePattern="${sys:pf.log.dir}/audit-sqlserver-failover.%d{yyyy-MM-dd}.log" >
<PatternLayout>
<pattern>%d | %X{event} | %X{subject} | %X{ip} | %X{app}| %X{host}| %X{protocol} | %X{role} | %X{connectionid} | %X{status}| %X{adapterid} | %X{description}| %X{responsetime} | %X{trackingid}| %n</pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy />
</Policies>
</RollingFile>
<PingFailover name="SecurityAuditToSQLServerDB-FAILOVER" primary="SecurityAuditToSQLServerDB" retryIntervalSeconds="60">
<Failovers>
<AppenderRef ref="SecurityAuditToSQLServerDB-FILE" />
</Failovers>
</PingFailover>
When I check the two different logs (by switching back & forth from the original config file) the database does not have as much data as the file did.
File:
2017-12-14 09:27:23,803| tid:_IJBevZYwaRBmmMzfZ1SZ1VdKtM| AUTHN_ATTEMPT| | 0:0:0:0:0:0:0:1 | | | | <Host name removed>| IdP| inprogress| LoginFormAdapter| | 505
DB:
1 2017-12-14 10:22:10.663 0
I don't know if it is relevant but I have also written a class to override the Jetty logs and they are also logging to the database and seem fine except for the user-agent field.
I suspect that I have interrupted some flow with what I have done but I have no idea what. Does anyone know why or where I am losing this data.

Answer 1

Instead of putting all those AppenderRef's in your Root section, try changing the appropriate Logger section for each type of logger (like IdpAuditLogger, SpAuditLogger, etc.). You may be running into a conflict there with multiple appenders loaded, as per the comment in the file:
Each must be the only appender referenced in the logger org.sourceid.websso.profiles.idp.IdpAuditLogger. If one of these appenders is added, remove the existing reference to "SecurityAudit2File".
And mentioned in the documentation here: https://support.pingidentity.com/s/document-item?bundleId=pingfederate-93&topicId=coj1564002980835.html
Important: As indicated in the IMPORTANT comments for the loggers, you must also remove some of the existing appender references.