Riak search on CRDT data types - memory backend - riak

I am using riak2.2.3, and trying to search in a map bucket type, but nothing is ever returned.
I've configured a bucket type "dist_cache" on the memory backend:
# riak-admin bucket-type status dist_cache
dist_cache is active
active: true
allow_mult: true
backend: <<"memory_mult">>
basic_quorum: false
big_vclock: 50
chash_keyfun: {riak_core_util,chash_std_keyfun}
claimant: 'riak#127.0.0.1'
datatype: map
dvv_enabled: true
dw: quorum
last_write_wins: false
linkfun: {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}
n_val: 3
notfound_ok: true
old_vclock: 86400
postcommit: []
pr: 0
precommit: []
pw: 0
r: quorum
rw: quorum
search_index: <<"expirable_token">>
small_vclock: 50
w: quorum
young_vclock: 20
I've then enable search in /etc/riak/:
search = on
Then I have configured an index, with the default schema and associated it with the bucket type (see above).
I can successfully store and retrieve values, using keys, in that bucket. I have stored 3 values in registers: binary data, integer (timestamp) and a string:
[
{{"attrs", :register}, <<131, 97, 111>>},
{{"iat_i", :register}, "1540923453"},
{{"test_s", :register}, "paul"}
]
(displayed after formatting from Elixir shell, using Elixir's Riak library.)
However, nothing is found when I try searching these values:
iex(74)> :riakc_pb_socket.search(pid, "expirable_token", "iat_i:[0 TO *]")
{:ok, {:search_results, [], 0.0, 0}}
iex(75)> :riakc_pb_socket.search(pid, "expirable_token", "iat_i:1540923453")
{:ok, {:search_results, [], 0.0, 0}}
iex(76)> :riakc_pb_socket.search(pid, "expirable_token", "test_s:paul")
{:ok, {:search_results, [], 0.0, 0}}
iex(77)> :riakc_pb_socket.search(pid, "expirable_token", "test_s:*")
{:ok, {:search_results, [], 0.0, 0}}
In addition, /var/log/riak/solr.log doesn't show any error message for these requests.
Am I missing something?
I needed to remove a few options from the java startup options, but now it seems java is up and running, and solr.log does show error message when trying malformed request.
EDIT:
After trying #vempo's solutions:
I have suffixed the field with _register, however it still does not work. Here is how the field is:
iex(12)> APISexAuthBearerCacheRiak.get("ddd", opts)
[
{{"attrs", :register}, <<131, 98, 0, 0, 1, 188>>},
{{"iat_i", :register}, "1542217847"},
{{"test_flag", :flag}, true},
{{"test_register", :register}, "pierre"}
]
but the search request still returns no result:
iex(15)> :riakc_pb_socket.search(pid, "expirable_token", "test_register:*")
{:ok, {:search_results, [], 0.0, 0}}
iex(16)> :riakc_pb_socket.search(pid, "expirable_token", "test_register:pierre")
{:ok, {:search_results, [], 0.0, 0}}
iex(17)> :riakc_pb_socket.search(pid, "expirable_token", "test_register:*")
{:ok, {:search_results, [], 0.0, 0}}
iex(18)> :riakc_pb_socket.search(pid, "expirable_token", "test_flag:true")
{:ok, {:search_results, [], 0.0, 0}}
iex(19)> :riakc_pb_socket.search(pid, "expirable_token", "test_flag:*")
Still know output in /var/log/riak/solr.log, and index seems correctly setup:
iex(14)> :riakc_pb_socket.list_search_indexes(pid)
{:ok,
[
[index: "expirable_token", schema: "_yz_default", n_val: 3],
[index: "famous", schema: "_yz_default", n_val: 3]
]}

For searching within maps the rules are different. According to Searching with Data Types, there are four schemas for maps, one for each embedded type:
*_flag
*_counter
*_register
*_set
So that in your case you should be searching attrs_register, iat_i_register, and test_s_register.
As a side note, the suffixes _s and _i are probably redundant. They are used by the default schema to decide the type of a regular field, but are useless with embedded datatypes).
UPDATE
And to sum up the rules:
a flag field named test will be indexed as test_flag (query test_flag:*)
a register field named test will be indexed as test_register (query test_register:*)
a counter field named test will be indexed as test_counter (query test_counter:*)
a set field named test will be indexed as test_set (query test_set:*)
This is nicely shown in the table in Searching with Data Types: Embedded Schemas.
See also the definition of dynamic fields for embedded datatypes in the default schema, section <!-- Riak datatypes embedded fields -->.

Related

dict(zip()) function compressing two lists in Python

These two lists imported from json file and I can't use dict(zip() properly.
l1 = ['grant_type', 'scope', 'client_id', 'client_id', 'grant_type', 'scope', 'code', 'redirect_uri']
l2 = ['client_credentials', 'AIS', '{{client_id}}', '{{client_id}}', 'authorization_code', '{{scope}}:{{consents_id}}', '{{response}}', '{{redirect_url}}']
lists = dict(zip(l1, l2))
print(lists)
Output:
{'grant_type': 'authorization_code', 'scope': '{{scope}}:{{consents_id}}', 'client_id': '{{client_id}}', 'code': '{{response}}', 'redirect_uri': '{{redirect_url}}'}
I guess there is something with double curly brackets "{{ }}", but I didn't find any information about them and how I can solve this problem.
Thank you for the response!

Get Shared channel list and joined user list via slack API

My goal is to list up Slack shared channel list, which joined users via Slack web API.
/usr/bin/curl -s -XPOST 'https://slack.com/api/conversations.list?token=MY_TOKEN&pretty=1' | jq -r '.channels[]|select(.is_shared = "true")'
But return is including non-shared channels also, like a (.is_shared = "false"). I have no idea why am I getting such results. Appreciate any help.
I use following code
/usr/bin/curl -s -XPOST 'https://slack.com/api/conversations.list?token=MY_TOKEN&pretty=1'
and results is
{
"ok": true,
"channels": [
{
"id": "C2U56FH6Z",
"name": "hoge_general",
"is_channel": true,
"is_group": false,
"is_im": false,
"created": 1477470814,
"is_archived": false,
"is_general": false,
"unlinked": 0,
"name_normalized": "hoge_general",
"is_shared": false,
"parent_conversation": null,
"creator": "U2UABCDEF",
"is_ext_shared": false,
"is_org_shared": false,
"shared_team_ids": [
"T2U94ABCDE"
],
"pending_shared": [],
"pending_connected_team_ids": [],
"is_pending_ext_shared": false,
"is_member": true,
"is_private": false,
"is_mpim": false,
"topic": {
"value": "Editor \2",
"creator": "U2UABCDE",
"last_set": 1478675694
},
"purpose": {
"value": "AAA editor ",
"creator": "U2UABABCDF",
"last_set": 14774596815
},
"previous_names": [],
"num_members": 11
},
So, I try to get channnel name which method is .
Comparison of values in jq is done using the == operator; = is the assignment operator which changes values (similar to many C-like programming languages).
The is_shared property in your JSON document is of type boolean (true or false), but you are comparing it to a string ("true" or "false"). That will always result in false (true == "true" is false).
Instead, compare booleans with booleans: .channels[] | select(.is_shared == true)
And, as with most other programming languages, conditionals compare implicitly against true, so if you have == true somewhere, this is redundant 99% of the time and can be simplified: .channels[] | select(.is_shared)

swi prolog 8.0.2 : gziped http

I tried to make work a piece of code that opens an http connection.
Nevertheless, web page may transfered as plain text or gziped.
As a result, the code with pragmatism tries to open as plain text and if it fails and receives an exception, tries as if it is gzip encoded.
URL is the sole variable to ground.
Try with URL = 'http://releases.llvm.org/6.0.0/tools/clang/docs/ClangCommandLineReference.html' for instance.
user::catch(
(
user::http_open(URL, DataStream, []),
user::load_html(stream(DataStream), Terms, []),
user::close(DataStream)
),
_
,
(
user::open_any(URL, read, GZipDataStream, CloseIt, [encoding(gzip), string(atom)]),
/*user::http:encoding_filter(gzip, DataStream, GZipDataStream),*/
user::load_html(stream(GZipDataStream), Terms, []),
user::close_any(CloseIt)
)
)
Infortunately, the recovery part of catch doesn't work.
Any suggestion, please ?
The user:: prefixes in the goals suggests that the code you posted is a fragment of Logtalk. If so, it's misusing Logtalk source code and creating a dependency on the SWI-Prolog autoloading mechanism. The code can be rewritten for clarity and resilience. Doing that and fixing the bug in it (library(zlib) must be loaded to make avaialble the http:encoding_filter/3 filter) results in the following solution:
:- use_module(library(http/http_open), []).
:- use_module(library(sgml), []).
:- use_module(library(iostream), []).
:- use_module(library(zlib), []).
:- object(html).
:- public(get_url/2).
% override ambiguous meta-predicate template
:- meta_predicate(sgml:load_html(*,*,*)).
get_url(URL, Terms) :-
catch(
setup_call_cleanup(
http:http_open(URL, DataStream, []),
sgml:load_html(stream(DataStream), Terms, []),
close(DataStream)
),
_,
setup_call_cleanup(
iostream:open_any(URL, read, DataStream, CloseIt, [string(atom)]),
sgml:load_html(stream(DataStream), Terms, []),
iostream:close_any(CloseIt)
)
).
:- end_object.
The setup_call_cleanup/3 calls ensure that the opened streams are closed in case of error.
Assuming the object above is saved in a html.lgt file, the following sample call shows it working for the URL you posted:
?- {html}.
...
% (0 warnings)
true.
?- html::get_url('http://releases.llvm.org/6.0.0/tools/clang/docs/ClangCommandLineReference.html', Terms).
Terms = [element(html, [xmlns='http://www.w3.org/1999/xhtml'], [element(head, [], [element(meta, ['http-equiv'='Content-Type', content='text/html; charset=utf-8'], []), element(title, [], ['Clang command line argument reference — Clang 6 documentation']), element(link, [... = ...|...], []), element(link, [...|...], []), element(..., ..., ...)|...]), element(body, [role=document], [' ', element(div, [... = ...|...], [element(..., ..., ...)|...]), '\n ', element(..., ..., ...)|...])])].

Ecto Changeset add functionality for warnings

I created a fork of ecto repository to extend Ecto.Changeset module with the ability to add warnings to the changeset. I wanted to have an add_warnings/4 function which adds a warning to the changeset as a simple list of warnings with this structure warnings: [{atom, {String.t, Keyword.t}}], similar to errors. The difference between the behavior of warnings and errors is that when an error occurs the data are not persisted, but when a warning occurs the data are persisted.
Ecto.Changeset struct extended with keys warnings and warningless?:
defstruct valid?: false, warningless?: false, data: nil, params: nil, changes: %{}, repo: nil,
errors: [], warnings: [], validations: [], required: [], prepare: [],
constraints: [], filters: %{}, action: nil, types: nil,
empty_values: #empty_values
Ecto functions for casting, changing, processing params, etc. adjusted. Function add_warnings/4 added:
#spec add_warning(t, atom, String.t, Keyword.t) :: t
def add_warning(%{warnings: warnings} = changeset, key, message, keys \\ []) when is_binary(message) do
%{changeset | warnings: [{key, {message, keys}}|warnings], warningless?: false}
end
The result is that I receive changeset with expected keys:
#Ecto.Changeset<action: nil, changes: %{}, data: #Company.Booking<>, errors: [],
valid?: true, warnings: [], warningless?: true>
When I make a change with error and warning I receive:
#Ecto.Changeset<action: nil,
changes: %{pickup_address: #Ecto.Changeset<action: :update,
changes: %{street_name: nil}, data: #Company.Address<>,
errors: [street_name: {"can't be blank", [validation: :required]}],
valid?: false,
warnings: [phone_number: {"This phone number is not common in Netherlands",
[]}], warningless?: false>}, data: #Company.Booking<>, errors: [],
valid?: false, warnings: [], warningless?: true>
So, everything is as expected, as far as warnings are concerned. Then, when I make a change with a warning but without an error, I receive:
#Ecto.Changeset<action: nil,
changes: %{pickup_address: #Ecto.Changeset<action: :update,
changes: %{street_name: "sss"}, data: #Company.Address<>, errors: [],
valid?: true,
warnings: [phone_number: {"This phone number is not common in Netherlands",
[]}], warningless?: false>}, data: #Company.Booking<>, errors: [],
valid?: true, warnings: [], warningless?: true>
Everything is as expected. When I don't make any changes to the form I still should receive a warning for phone number, but I receive:
#Ecto.Changeset<action: nil, changes: %{}, data: #Company.Booking<>, errors: [],
valid?: true, warnings: [], warningless?: true>
I got a changeset without any warnings as there is no changes key in changeset because the data didn't change.
The question is as follows, how to implement warnings functionality to always have warnings in the changeset, even if no change was made?
You should consider to prefill the warnings at the very beginning of the each changeset function you would create - since you can't use plug there you can come up to write a macro that will handle this logic for you, __using__ is advised, so it would be quite easy to distinguish your logic from Ecto's default logic.
Your validation shouldn't add warnings to warnings list, but you have to implement it another way around - if the field is fine, you would remove already existing warnings from this list. That way you would be sure that your changeset is fine when it's warningless, because it removed all the warnings from this list and it would works perfectly for empty changes in changeset.

Cannot find function `jsx:is_json/1` although jsx is included and compiled

I am writing a webapp using Erlang toolchain (OTP, rebar3, cowboy, jsx...). The following code does not work because jsx:is_json/1 cannot be found during runtime.
handle_login(Req, State) ->
{ok, Data, _} = cowboy_req:body(Req),
case jsx:is_json(Data) of
false -> cowboy_req:reply(400,
[
{<<"content-type">>, <<"application/json">>}
],
<<"Invalid JSON">>,
Req);
Stacktrace:
{[{reason,undef},
{mfa,{erbid_api_handler,handle,2}},
{stacktrace,
[{jsx,is_json,[<<"{\"username\":\"tom\"}">>],[]},
{erbid_api_handler,handle_login,2,
[{file,
"/Users/khanhhua/dev/project-erbid/_build/default/lib/erbid/src/erbid_api_handler.erl"},
{line,45}]},
{erbid_api_handler,handle,2,
... truncated for brevity
Folder structure:
I need to know how to fix the issue. Thanks.
I have found the cause of this issue. I did not not include module jsx in my erbid.app.src's applications section.
{application, erbid, [
{description, "Realtime system"},
{vsn, "0.1.0"},
{registered, []},
{applications, [
kernel,
stdlib,
cowboy,
jsx
]},
{mod, {erbid, []}},
{env, []}
]}.
Totally due to my lack of Erlang experience.

Resources