I have the following query
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"" OR =ecmcnddoc:doc_name_ru:"\-") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
and it has different results in Alfresco NodeBrowser (fts-alfresco) and in Java code "solr-fts-alfresco" (both results should be < 1000 and they are)
SearchParameters searchParameters = new SearchParameters();
searchParameters.setLanguage(SearchService.LANGUAGE_SOLR_FTS_ALFRESCO);
searchParameters.addStore(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE);
searchParameters.setLimitBy(LimitBy.UNLIMITED);
searchParameters.setLimit(1000);
searchParameters.setPermissionEvaluation(PermissionEvaluationMode.EAGER);
searchParameters.addLocale(new Locale("ru", "RU"));
searchParameters.setQuery(query);
tempResultSet = customSearchService.query(searchParameters);
Also, in the Java code the clause =ecmcnddoc:doc_name_ru:""may fails two different ways: as always FALSE in query:
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"" OR =ecmcnddoc:doc_name_ru:"\-") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
And always TRUE in query:
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
Could you tell me the proper way to use =ecmcnddoc:doc_name_ru:"" clause?
Thank you!
Assuming you're looking for results that have an empty "doc_name_ru", have you tried:
-EXISTS:"ecmcnddoc:doc_name_ru"
As I understand, the Chrome browser uses the WebKit time format for timestamps within the browser history database. WebKit time is expressed as milliseconds since January, 1601.
I've found numerous articles that seemingly have the answer to my question, but none have worked so far. The common answer is to use the formula below to convert from WebKit to a human-readable, localtime:
SELECT datetime((time/1000000)-11644473600, 'unixepoch', 'localtime') AS time FROM table;
Sources:
https://linuxsleuthing.blogspot.com/2011/06/decoding-google-chrome-timestamps-in.html
What is the format of Chrome's timestamps?
I'm trying to convert the timestamps while gathering the data through Osquery, using the configuration below.
"chrome_browser_history" : {
"query" : "SELECT urls.id id, urls.url url, urls.title title, urls.visit_count visit_count, urls.typed_count typed_count, urls.last_visit_time last_visit_time, urls.hidden hidden, visits.visit_time visit_time, visits.from_visit from_visit, visits.visit_duration visit_duration, visits.transition transition, visit_source.source source FROM urls JOIN visits ON urls.id = visits.url LEFT JOIN visit_source ON visits.id = visit_source.id",
"path" : "/Users/%/Library/Application Support/Google/Chrome/%/History",
"columns" : ["path", "id", "url", "title", "visit_count", "typed_count", "last_visit_time", "hidden", "visit_time", "visit_duration", "source"],
"platform" : "darwin"
}
"schedule": {
"chrome_history": {
"query": "select distinct url,datetime((last_visit_time/1000000)-11644473600, 'unixepoch', 'localtime') AS time from chrome_browser_history where url like '%nhl.com%';",
"interval": 10
}
}
The resulting events have timestamps from the year 1600:
"time":"1600-12-31 18:46:16"
If I change the config to pull the raw timestamp with no conversion, I get stamps such as the following:
"last_visit_time":"1793021894"
From what I've read about WebKit time, it is expressed in 17-digit numbers, which clearly is not what I'm seeing. So I'm not sure if this is an Osquery, Chrome, or query issue at this point. All help and insight appreciated!
Solved. The datetime conversion needs to take place within the table definition query.
I.e. the query defined underneath "chrome_browser_history".
"chrome_browser_history" : {
"query" : "SELECT urls.id id, urls.url url, urls.title title, urls.visit_count visit_count, urls.typed_count typed_count, datetime(urls.last_visit_time/1000000-11644473600, 'unixepoch') last_visit_time, urls.hidden hidden, visits.visit_time visit_time, visits.from_visit from_visit, visits.visit_duration visit_duration, visits.transition transition, visit_source.source source FROM urls JOIN visits ON urls.id = visits.url LEFT JOIN visit_source ON visits.id = visit_source.id",
"path" : "/Users/%/Library/Application Support/Google/Chrome/%/History",
"columns" : ["path", "id", "url", "title", "visit_count", "typed_count", "last_visit_time", "hidden", "visit_time", "visit_duration", "source"],
"platform" : "darwin"
}
"schedule": {
"chrome_history": {
"query": "select distinct url,last_visit_time from chrome_browser_history where url like '%nhl.com%';",
"interval": 10
}
}
Trying to make the conversion within the osquery scheduled query (as I was trying before) will not work. i.e:
"schedule": {
"chrome_history": {
"query": "select distinct url,datetime((last_visit_time/1000000)-11644473600, 'unixepoch', 'localtime') AS time from chrome_browser_history where url like '%nhl.com%';",
"interval": 10
}
}
Try:
SELECT datetime(last_visit_time/1000000-11644473600, \"unixepoch\") as last_visited, url, title, visit_count FROM urls;
This is from something I wrote up a while ago - One-liner that runs osqueryi with ATC configuration to read in the chrome history file, export as json and curl the json to an API endpoint
https://gist.github.com/defensivedepth/6b79581a9739fa316b6f6d9f97baab1f
The things you're working with, are pretty straight sqlite. So I would start by debugging inside sqlit.
First, you should verify the data is what you expect. On my machine, I see:
$ cp Library/Application\ Support/Google/Chrome/Profile\ 1/History /tmp/
$ sqlite3 /tmp/History "select last_visit_time from urls limit 2"
13231352154237916
13231352154237916
Second, I would verify the underlying math:
sqlite> select datetime(last_visit_time/1000000-11644473600, "unixepoch") from urls limit 2;
2020-04-14 15:35:54
2020-04-14 15:35:54
It would be easier to test your config snippet if you included it as text we can copy/paste.
So say I am using a form to build a query against my datasource (i've come so far in two weeks! I can do this!), how do I make it more complex?
What if I want books by austen that include the word "pride" AND books by gabaldon that contain the word "Snow"
the individual queries would be
widget.datasource.query.filters['author']._contains = "austen";
widget.datasource.query.filters['title']._contains = "pride";
and
widget.datasource.query.filters['author']._contains = "gabaldon";
widget.datasource.query.filters['title']._contains = "snow";
in pseudosql it would be
select * from table
where
((author like 'austen') and (title like 'snow'))
or
((author like 'gabaldon') and (title like 'pride'))
Is there a way to filter a data source on a complex query like this and cut out the whole widget.datasource aspect? I'd be fine with using a calculated table.
Edit: Ok i'm making some progress towards the kind of functionality I need, can anyone tell me why this works:
widget.datasource.query.filters.document_name._contains = 'x';
but this does not?
widget.datasource.query.parameters.v1 = "x";
widget.datasource.query.where = 'document_name contains :v1';
this also doesn't work:
widget.datasource.query.where = 'document_name contains "x"';
I have a search request written as
import sqlite3
conn = sqlite3.connect('locker_data.db')
c = conn.cursor()
def search1(teacher):
test = 'SELECT Name FROM locker_data WHERE Name or Email LIKE "%{0}%"'.format(teacher)
data1 = c.execute(test)
return data1
def display1(data1):
Display1 = []
for Name in data1:
temp1 = str(Name[0])
Display1.append("Name: {0}".format(temp1))
return Display1
def locker_searcher(teacher):
data = display1(search1(teacher))
return data
This allows me to search for the row containing "Mr FishyPower (Mr Swag)" or "Mr FishyPower / Mr Swag" with a search input of "FishyPower". However, when I try searching with an input of "Swag", I am then unable to find the same row.
In the search below, it should have given me the same search results.
The database is just a simple 1x1 sqlite3 database containing 'FishyPower / Mr Swag'
Search Error on 'Swag'
Edit: I technically did solve it by limiting the columns being searched to only 'Name' but I intended the code search both the 'Name' and 'Email' columns and output the results as long as the search in within either or both columns.
Edit2: SELECT Name FROM locker_data WHERE Email LIKE "%{0}%" or Name LIKE "%{0}%" was the right way to go.
I'm gonna guess that Mr. FishyPower's email address is something like mrFishyPower#something.com. The query is only comparing Email to teacher. If it was
WHERE Name LIKE "%{0}%"
OR Email LIKE "%{0}%"'
you would (probably) get the result you want.
Say I have the following documents in my database: a_doc1, a_doc2, b_doc1, and b_doc2
All these documents are of the following format
<doc>
....
<updatedTime>2011-02-07T14:41:02.133-05:00</updatedTime>
....
</doc
>
The value of the "updatedTime" element is inserted when the document is created using the fn:current-dateTime()
Now I am trying to do the following:
find all documents whose name starts with "a_"
order these documents by their <updatedTime> element in descending order
Return the first document name from the descending order
I tried the following:
for $doc_name in db:list()
where fn:starts-with($doc_name, 'a_')
order by xs:dateTime(doc($doc_name)/updatedTime) descending
return $doc_name
Say "a_doc1" is created at "2011-02-07T14:40:00.78-05:00" and "a_doc2" is created at "2011-02-07T14:41:02.133-05:00", the desired output is a_doc2. In short the document name(starting with a_) of the most recent document created must be returned.
When I try my sample code, the output returned is : [a_doc1, a_doc2].
The expected output is: [a_doc2, a_doc1].
Thanks,
Sony
order by dateTime(doc($doc_name)/updatedTime/text())
Instead of this use:
order by xs:dateTime(doc($doc_name)/updatedTime)
you should use (xs: http://www.w3.org/2001/XMLSchema):
xs:dateTime($arg as xs:anyAtomicType?) as xs:dateTime?
Example:
xs:dateTime("1999-12-31T12:00:00")
instead of (fn: http://www.w3.org/2005/xpath-functions)
fn:dateTime($arg1 as xs:date?, $arg2 as xs:time?) as xs:dateTime?
Example:
fn:dateTime(xs:date("1999-12-31"), xs:time("12:00:00"))