I have an nginx server using syslog-ng to send access and error logs to a remote syslog-ng server. I am having it tag the messages so that the remote server can filter on the tags to put them into files. But the filter statements seem to be not working. On the local client I did a test, sending the messages to a local file using the filters and they work there. But they seem to break somehow when being sent remote.
The config on the client is:
#version: 3.13
#include "scl.conf"
## global options.
options { chain_hostnames(off);
flush_lines(0);
use_dns(no);
use_fqdn(no);
owner("root");
group("adm");
perm(0640);
stats_freq(0);
bad_hostname("^gconfd$");
};
source s_qa_nginx_access {
file("/var/log/nginx/access.log" follow-freq(1)
tags("qa_nginx_access")
flags(no-parse));
};
source s_qa_nginx_error {
file("/var/log/nginx/error.log" follow-freq(1)
tags("qa_nginx_error")
flags(no-parse));
};
destination d_syslog-ng_central { syslog("10.0.0.50" transport("tcp") port(514)); };
log { source(s_qa_nginx_access); destination(d_syslog-ng_central);};
log { source(s_qa_nginx_error); destination(d_syslog-ng_central);};
On the remote syslog-ng server I have
#version: 3.13
#include "scl.conf"
options {
flush_lines(0);
use_dns(no);
use_fqdn(no);
owner("root");
group("adm");
perm(0640);
stats_freq(0);
bad_hostname("^gconfd$");
time-reap(30);
mark-freq(10);
keep-hostname(yes);
};
source s_network { syslog(transport(tcp) port(514)); };
filter f_qa_nginx_access { tags("qa_nginx_access"); };
filter f_qa_nginx_error { tags("qa_nginx_error"); };
destination d_qa_nginx_access {
file(
"/var/log/remote/qa_nginx_access.log"
owner("root")
group("adm")
perm(0640)
);
};
destination d_qa_nginx_error {
file(
"/var/log/remote/qa_nginx_error.log"
owner("root")
group("adm")
perm(0640)
);
};
log { source(s_network); filter(f_qa_nginx_access); destination(d_qa_nginx_access); };
log { source(s_network); filter(f_qa_nginx_error); destination(d_qa_nginx_error); };
If I remove the filter from the log statement all of the log messages go to both files. but with the filter in place nothing makes it to any of the files on the remote server. Is it somehow not sending the tags to remote?
You might want to refer to syslog-ng administration guide. Below are some of the important notes from the guide. If you need to send the tags remotely, use SDATA.meta.tags instead or you can use the template to write is as part of the message too.
Full admin guide can be find at the following link:
https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/58
Tags are available locally, that is, if you add tags to a message on the client, these tags will not be available on the server.
To include the tags in the message, use the ${TAGS} macro in a template. Alternatively, if you are using the IETF-syslog message format, you can include the ${TAGS} macro in the .SDATA.meta part of the message. Note that the ${TAGS} macro is available only in syslog-ng OSE 3.1.1 and later.
While creating the connection from NetSuite to SFTP using N/SFTP module i'm facing an error states:
"FTP_INCORRECT_HOST_KEY","message":"Provided host key does not match
remote server's fingerprint."
I have tried checking with my server team but no hope. Can any one suggest me how to resolve this or how can i get an authorized finger print host key from server.
I have tried with Suitescript 2.0 module (N/SFTP) with the help of the tool mentioned below.
https://ursuscode.com/netsuite-tips/suitescript-2-0-sftp-tool/
/**
*#NApiVersion 2.x
#NScriptType ScheduledScript
*/
define(['N/sftp', 'N/file', 'N/runtime'],function(sftp, file,runtime) {
function execute(context)
{
var myPwdGuid = "Encrypted password by GUID";
var myHostKey = "Some long Host key around 380 characters";
// establish connection to remote FTP server
var connection = sftp.createConnection({
username: 'fuel_integration',
passwordGuid: myPwdGuid, // references var myPwdGuid
url: '59.165.215.45',//Example IP
directory: '/sftproot/TaleoSync',
restrictToScriptIds : runtime.getCurrentScript().id,
restrictToCurrentUser :false,
hostKey: myHostKey // references var myHostKey
});
// specify the file to upload using the N/file module
// download the file from the remote server
var downloadedFile = connection.download({
directory: '/sftproot/TaleoSync',
filename: 'Fuel Funnel Report_without filter.csv'
});
downloadedFile.folder = ;
downloadedFile.save();
context.response.write(' Downloaded "Fuel Funnel Report_without filter" to fileCabinet');
}
return {
execute: execute
};
});
I expect to create a connection between SFTP and NetSuite to down a file from SFTP and place it to NetSuite file cabinet.
A couple of things:
restrictToScriptIds : runtime.getCurrentScript().id,
restrictToCurrentUser :false,
Are not part of the createConnection signature. Those should have been used when you created a Suitelet to vault your credential.
However the hostkey complaint may be dealt with by using ssh-keyscan from a linux box.
ssh-keyscan 59.165.215.45
should replay with the server name then ssh-rsa then a long base64 string. Copy that string so it ends up in myHostKey and set the hostKeyType to RSA.
I'm using Retrofit (with RxJava) to download a text file.
#Streaming
#GET
Observable<ResponseBody> download(#Url String file);
Everything works fine except when user is using a public wifi that need web authentication. In this case download is still successful and what I receive is a file contains HTML of that authentication web page. How do I check and raise error in this case? I tried with Interceptor, NetworkInterceptor (Okhttp), disable redirect (Retrofit) but no luck because no exception is thrown.
Either checksum your file and check it when you get it, or check the Content-type:
#Streaming
#GET
Observable<Response<ResponseBody>> download(#Url String file);
download("http://...")
.map(response -> {
if(!"application/my-content-type".equals(
response.headers("Content-Type")) {
throw new RuntimeException("Bad download");
}
return resource.body();
})
I am using Xcode 7.3 for iOS 9.3 to try and encrypt a Core Data file. I am trying to use NSPersistentStoreFileProtectionKey and set it to NSFileProtectionComplete to enable the encryption. It is not working for some reason and I can always see the .sqlite file generated by the app and browse through the content in sqlitebrowser or iexplorer. Here is my code :
lazy var persistentStoreCoordinator: NSPersistentStoreCoordinator = {
// The persistent store coordinator for the application. This implementation creates and returns a coordinator, having added the store for the application to it. This property is optional since there are legitimate error conditions that could cause the creation of the store to fail.
// Create the coordinator and store
let coordinator = NSPersistentStoreCoordinator(managedObjectModel: self.managedObjectModel)
let url = self.applicationDocumentsDirectory.URLByAppendingPathComponent("SingleViewCoreData.sqlite")
var failureReason = "There was an error creating or loading the application's saved data."
let dict: [NSObject : AnyObject] = [
NSPersistentStoreFileProtectionKey : NSFileProtectionComplete
]
do {
try coordinator.addPersistentStoreWithType(NSSQLiteStoreType, configuration: nil, URL: url, options: dict)
} catch {
// Report any error we got.
var dict = [String: AnyObject]()
dict[NSLocalizedDescriptionKey] = "Failed to initialize the application's saved data"
dict[NSLocalizedFailureReasonErrorKey] = failureReason
dict[NSUnderlyingErrorKey] = error as NSError
let wrappedError = NSError(domain: "YOUR_ERROR_DOMAIN", code: 9999, userInfo: dict)
// Replace this with code to handle the error appropriately.
// abort() causes the application to generate a crash log and terminate. You should not use this function in a shipping application, although it may be useful during development.
NSLog("Unresolved error \(wrappedError), \(wrappedError.userInfo)")
abort()
}
do {
let url = self.applicationDocumentsDirectory.URLByAppendingPathComponent("SingleViewCoreData.sqlite")
try NSFileManager.defaultManager().setAttributes([NSFileProtectionKey : NSFileProtectionComplete], ofItemAtPath: url.path!)
} catch {
}
do {
let url = self.applicationDocumentsDirectory.URLByAppendingPathComponent("SingleViewCoreData.sqlite-wal")
try NSFileManager.defaultManager().setAttributes([NSFileProtectionKey : NSFileProtectionComplete], ofItemAtPath: url.path!)
// try print(NSFileManager.defaultManager().attributesOfFileSystemForPath(String(url)))
} catch {
}
do {
let url = self.applicationDocumentsDirectory.URLByAppendingPathComponent("SingleViewCoreData.sqlite-shm")
try NSFileManager.defaultManager().setAttributes([NSFileProtectionKey : NSFileProtectionComplete], ofItemAtPath: url.path!)
// try print(NSFileManager.defaultManager().attributesOfFileSystemForPath(String(url)))
} catch {
}
return coordinator
}()
I have also enabled Data Protection for my target in the "Capabilities". I have regenerated the provisioning profile from the Apple Developer portal and am using that with Enabled Data Protection.
I am also using the following code to check the file attributes of .sqlite , .sqlite-wal and .sqlite-shm files. NSFileProtectionKey is correctly set for all 3 of them.
func checkProtectionForLocalDb(atDir : String){
let fileManager = NSFileManager.defaultManager()
let enumerator: NSDirectoryEnumerator = fileManager.enumeratorAtPath(atDir)!
for path in enumerator {
let attr : NSDictionary = enumerator.fileAttributes!
print(attr)
}
}
I also tried disabling the Journal mode to prevent -wal and -shm files from being created. But I can still read the .sqlite file. Even though the attributes read NSFileProtectionComplete.
As described in the Apple Documentation at Apple Docs under "Protecting Data using On Disk Encryption", I tried to check whether the value of variable protectedDataAvailable changes as shown in the code below
public func applicationDidEnterBackground(application: UIApplication) {
// Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later.
// If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits.
NSThread.sleepForTimeInterval(10)
sleep(10)
let dataAvailable : Bool = UIApplication.sharedApplication().protectedDataAvailable
print("Protected Data Available : " + String(dataAvailable))
}
If I check the value without the delay it's set to true but after adding the delay it's set to false. This is kind of encouraging, however, right after, when I download the container, to show the content, it still has .sqlite file that still shows the content when opened in sqlitebrowser.
Ok, I finally understand this.
Using Xcode 7.3.1
Enabling File Protection
Enable File Protection using the Capabilities tab on your app target
If you do not want the default NSFileProtectionComplete, change this setting in the developer portal under your app id
Make sure Xcode has the new provisioning profile this creates.
For protecting files your app creates, that's it.
To protect Core Data, you need to add the NSPersistentStoreFileProtectionKey: NSFileProtectionComplete option to your persistent store.
Example:
var options: [NSObject : AnyObject] = [NSMigratePersistentStoresAutomaticallyOption: true,
NSPersistentStoreFileProtectionKey: NSFileProtectionComplete,
NSInferMappingModelAutomaticallyOption: true]
do {
try coordinator!.addPersistentStoreWithType(NSSQLiteStoreType, configuration: nil, URL: url, options: options)
Testing File Protection
I am not able to test this using a non-jailbroken device connected to a computer. Every attempt to access the device this way requires that I "trust" the computer and I believe that trusted computers are always able to read the phone's data ("Trusted computers can sync with your iOS device, create backups, and access your device's photos, videos, contacts, and other content" - https://support.apple.com/en-us/HT202778). I think the other answers on SO referencing this technique are no longer valid with more recent versions of iOS. Indeed, I am always able to download the container using XCode and view the app's data using iPhone Explorer. So how to test...
1 - Create an archive and ensure that it is has the proper entitlements by running the following on the .app file from the command line:
codesign -d --entitlements :- <path_to_app_binary>
You should see a key/value pair that represents your Data Protection level. In this example, NSFileProtectionComplete:
<key>com.apple.developer.default-data-protection</key>
<string>NSFileProtectionComplete</string>
In addition, I used the following two techniques to satisfy myself that the data protection is indeed working. They both require code changes.
2 - Add some code to verify that the proper NSFileProtectionKey is being set on your files and/or core data store:
NSFileManager.defaultManager().attributesOfItemAtPath(dbPath.path!)
If I print this out on one of my files I get:
["NSFileCreationDate": 2016-10-14 02:06:39 +0000, "NSFileGroupOwnerAccountName": mobile, "NSFileType": NSFileTypeRegular, "NSFileSystemNumber": 16777218, "NSFileOwnerAccountName": mobile, "NSFileReferenceCount": 1, "NSFileModificationDate": 2016-10-14 02:06:39 +0000, "NSFileExtensionHidden": 0, "NSFileSize": 81920, "NSFileGroupOwnerAccountID": 501, "NSFileOwnerAccountID": 501, "NSFilePosixPermissions": 420, "NSFileProtectionKey": NSFileProtectionComplete, "NSFileSystemFileNumber": 270902]
Note the "NSFileProtectionKey": "NSFileProtectionComplete" pair.
3 - Modify the following code and hook it up to some button in your app.
#IBAction func settingButtonTouch(sender: AnyObject) {
updateTimer = NSTimer.scheduledTimerWithTimeInterval(0.5, target: self,
selector: #selector(TabbedOverviewViewController.runTest), userInfo: nil, repeats: true)
registerBackgroundTask()
}
var backgroundTask: UIBackgroundTaskIdentifier = UIBackgroundTaskInvalid
var updateTimer: NSTimer?
func registerBackgroundTask() {
backgroundTask = UIApplication.sharedApplication().beginBackgroundTaskWithExpirationHandler {
[unowned self] in
self.endBackgroundTask()
}
assert(backgroundTask != UIBackgroundTaskInvalid)
}
func endBackgroundTask() {
NSLog("Background task ended.")
UIApplication.sharedApplication().endBackgroundTask(backgroundTask)
backgroundTask = UIBackgroundTaskInvalid
}
func runTest() {
switch UIApplication.sharedApplication().applicationState {
case .Active:
NSLog("App is active.")
checkFiles()
case .Background:
NSLog("App is backgrounded.")
checkFiles()
case .Inactive:
break
}
}
func checkFiles() {
// attempt to access a protected resource, i.e. a core data store or file
}
When you tap the button this code begins executing the checkFiles method every .5 seconds. This should run indefinitely with the app in the foreground or background - until you lock your phone. At that point it should reliably fail after roughly 10 seconds - exactly as described in the description of NSFileProtectionComplete.
We need to understand how Data Protection works.
Actually, you don't even need to enable it. Starting with iOS7, the default protection level is “File Protection Complete until first user authentication.”
This means that the files are not accessible until the user unlocks the device for the first time. After that, the files remain accessible even when the device is locked and until it shuts down or reboots.
The other thing is that you're going to see the app's data on a trusted computer always - regardless of the Data Protection level setting.
However, the data can’t be accessed if somebody tries to read them from the flash drive directly. The purpose of Data Protection is to ensure that sensitive data can’t be extracted from a password-protected device’s storage.
After running this code, I could still access and read the contents written to protectedFileURL, even after locking the device.
do {
try data.write(to: protectedFileURL, options: .completeFileProtectionUnlessOpen)
} catch {
print(error)
}
But that's normal since I ran iExplorer on a trusted computer.
And for the same reason, it's fine if you see your sqlite file.
The situation is different if your device gets lost or stolen. A hacker won't be able to read the sqlite file since it's encrypted. Well, unless he guesses your passcode somehow.
Swift 5.0 & Xcode 11:
Enable "Data Protection" in "Capabilities".
Use the following code to protect a file or folder at a specific path:
// Protects a file or folder + excludes it from backup.
// - parameter path: Path component of the file.
// - parameter fileProtectionType: `FileProtectionType`.
// - returns: True, when protected successful.
static func protectFileOrFolderAtPath(_ path: String, fileProtectionType: FileProtectionType) -> Bool {
guard FileManager.default.fileExists(atPath: path) else { return false }
let fileProtectionAttrs = [FileAttributeKey.protectionKey: fileProtectionType]
do {
try FileManager.default.setAttributes(fileProtectionAttrs, ofItemAtPath: path)
return true
} catch {
assertionFailure("Failed protecting path with error: \(error).")
return false
}
}
(Optional) Use the following code to check whether the file or folder at the specific path is protected (note: This only works on physical devices):
/// Returns true, when the file at the provided path is protected.
/// - parameter path: Path of the file to check.
/// - note: Returns true, for simulators. Simulators do not have hardware file encryption. This feature is only available for real devices.
static func isFileProtectedAtPath(_ path: String) -> Bool {
guard !Environment.isSimulator else { return true } // file protection does not work on simulator!
do {
let attributes = try FileManager.default.attributesOfItem(atPath: path)
if attributes.contains(where: { $0.key == .protectionKey }) {
return true
} else {
return false
}
} catch {
assertionFailure(String(describing: error))
return false
}
}
Rather than encrypt a file at the local level I set NSFileProtectionComplete for the app as a whole.
Create the file 'entitlements.plist' in your apps root folder with the following content.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>DataProtectionClass</key>
<string>NSFileProtectionComplete</string>
</dict>
</plist>
Then if you haven't already done so already (this could be the problem with your file level encryption) enable Data Protection in your apps capabilities.
I have looked everywhere and I can't find out how to do this; I'm so frustrated...
How can I allow the user to send (via email) the SQLite db file?
That's it in a nutshell. I can convert it to string and attach, but I want to send the actual db file. And I'm using a new phone that doesn't have an external SD card.
The app is just a form that the user fills out, then it's saved to a SQLite database. That works wonderfully. As does printing the db to string (text) and then sending it. But, I want the user to email the actual db file (so I can use C# to read, process it, and "recreate" a real form).
Or should I be using something other than SQLite?
Edit: This is as far as I've made it. It seems to work, but it does not actually attach the file or rather the file is "blank/empty". Debug log says no such file or directory. screenshot of debug log here:http://imgur.com/oyzdtuJ
//trying again to send a SQL db file
//this seems to work and shows that it's attaching a file, but the file is empty so it won't attach
//gmail will say "cant attach empty file"
private void sendFile(String email){
File myFile = this.getFileStreamPath("testresults.db");
if(myFile != null) {
Log.d("LOG PRINT SHARE DB", "File Found, Here is file location: " + myFile.toString());
}else {
Log.w("Tag", "file not found!");
}
Uri contentUri = FileProvider.getUriForFile(this, "com.columbiawestengineering.columbiawest.MainActivity", myFile);
Log.d("LOG PRINT SHARE DB", "contentUri got: here is contentUri: " + contentUri.toString());
//grant permision for app with package "com.columbiawestengineering.columbiawest", eg. before starting other app via intent
this.grantUriPermission("com.columbiawestengineering.columbiawest", contentUri, Intent.FLAG_GRANT_WRITE_URI_PERMISSION | Intent.FLAG_GRANT_READ_URI_PERMISSION);
Log.d("LOG PRINT SHARE DB", "permission granted, here is contentUri: " + contentUri.toString());
Intent shareIntent = new Intent();
shareIntent.setAction(Intent.ACTION_SEND);
shareIntent.setType("application/octet-stream");
shareIntent.putExtra(Intent.EXTRA_SUBJECT, "blaaa subject");
String to[] = { email };
shareIntent.putExtra(Intent.EXTRA_EMAIL, to);
shareIntent.putExtra(Intent.EXTRA_TEXT, "blah blah message");
shareIntent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION);
shareIntent.putExtra(Intent.EXTRA_STREAM, contentUri);
startActivityForResult(Intent.createChooser(shareIntent, "Send mail..."), 1252);
//revoke permisions
this.revokeUriPermission(contentUri, Intent.FLAG_GRANT_WRITE_URI_PERMISSION | Intent.FLAG_GRANT_READ_URI_PERMISSION);
}
See this answer
Android Utility to send sqlite db to server
You could do this any number of ways. I'd say posting it to a web service is easiest. If you can only use email then I'd compress and encode it and attach it to an email but that sounds painful.
Solved. FileProvider cannot access the database directory. The db file must be copied to the files directory before it is attached. See solution here:Android: FileProvider "Failed to find configured root"