getting values from dictionary with clickhouse database - dictionary

I am learning Clickhouse databse and I have a challenge with some lab:
I have a dictionary with the following structure / schema:
CREATE DICTIONARY reputation_rbl_db (  network String,  source String,  category String ,asn UInt32,country String ) PRIMARY KEY network SOURCE(FILE(  path '/var/lib/clickhouse/user_files/rbl_database.csv'  format 'CSVWithNames' )) LAYOUT(IP_TRIE()) LIFETIME(300);
I have a lot of rows like:
┌─network────────┬─source────┬─category────┬───asn─┬─country─┐
│ 1.1.109.250/32 │ blacklist │ rbl │ 2519 │ JP │
│ 1.10.16.0/20 │ blacklist │ rb │ 22 │ CN │
│ 1.12.45.0/24 │ blacklist │ rbl │ 45090 │ CN │
└────────────────┴───────────┴─────────────┴───────┴─────────┘
I need to find the SELECT which search if the IP 1.10.15.55 is under some network with CIDR at the dictionary, something to
SELECT * from reputation_rbl_db where network= '1.10.15.55/32'
and the response needs to be something like:
1.10.16.0/20, blacklist, rbl, 22, CN
is there any way to find this select ?
Thanks in advance
Max
i tried with:
SELECT * FROM dictionary('reputation_rbl_db') where network = '1.10.15.55/32' but do not work because the select not find the network which has this IP

IP Address 1.10.15.55 is NOT in CIDR range 1.10.16.0/20 ;).
Anyway,
There is a IPv4CIDRToRange function in Clickhouse:
SELECT IPv4CIDRToRange(toIPv4('1.10.16.0'), 20)
Query id: 598aa84f-6297-4d99-8b0c-29a571370024
┌─IPv4CIDRToRange(toIPv4('1.10.16.0'), 20)─┐
│ ('1.10.16.0','1.10.31.255') │
└──────────────────────────────────────────┘
You can also use Clickhouse bit functions (or bit operators in other DBMSs):
SELECT
CAST(CAST('255.255.240.0', 'IPv4'), 'UInt32') AS mask,
CAST(bitAnd(CAST(CAST('1.10.16.0', 'IPv4'), 'UInt32'), mask), 'IPv4') AS lower,
CAST(bitOr(lower, bitNot(CAST(mask, 'UInt32'))), 'IPv4') AS upper
┌───────mask─┬─lower─────┬─upper───────┐
│ 4294963200 │ 1.10.16.0 │ 1.10.31.255 │
└────────────┴───────────┴─────────────┘
Or you can just utilize isIPAddressInRange function (easy way):
SELECT isIPAddressInRange('1.10.15.55', '1.10.16.0/20')
┌─isIPAddressInRange('1.10.15.55', '1.10.16.0/20')─┐
│ 0 │
└──────────────────────────────────────────────────┘
Example:
SELECT * FROM reputation_rbl_db
WHERE isIPAddressInRange(extract('1.10.15.55/32', '[^/]*'), network)

Related

Facing error with terraform code for importing multiple runbook to azure automation account

I have created the azure automation account using terraform code. I have multiple runbooks PowerShell scripts saved in my local. I am using for.each option to import all the runbooks at a time. But I am getting some errors while running the terraform file. Please find my code below:
resource "azurerm_automation_runbook" "example" {
for_each = fileset(".", "./Azure_Runbooks/*.ps1")
name = ["${split("/", each.value)}"][1]
location = var.location
resource_group_name = var.resource_group
automation_account_name = azurerm_automation_account.example.name
log_verbose = var.log_verbose
log_progress = var.log_progress
runbook_type = var.runbooktype
content = filemd5("${each.value}")
}
Error:
Error: Invalid index
│
│ on AutomationAccount\main.tf line 51, in resource "azurerm_automation_runbook" "example":
│ 51: name = ["${split("/", each.value)}"][1]
│ ├────────────────
│ │ each.value will be known only after apply
│
│ The given key does not identify an element in this collection value: the given index is greater than
│ or equal to the length of the collection.
Can someone please help how I can upload all my existing runbook scrips to the newly created automation account using terraform code.
You don't need list in a list. So instead of
name = ["${split("/", each.value)}"][1]
it should be
name = split("/", each.value)[1]

Javascript bridge / upcall to JavaFX (via JSObject.setMember() method) breaks when moving class into a package

This is similar to this question, but I believe we are encountering different issues.
Setup:
I have a Kotlin class that interfaces with a TinyMCE instance running in a JavaFX Webview. I am setting up upcalls from Javascript to JavaFX as shown below, in the TinyMCEInterface class:
inner class BridgeObject {
fun setEditorAndSelection(ed : JSObject?) {
editorObj = ed
selectionObj = editorObj?.getMember("selection") as JSObject?
}
fun setInterfaceContent(newContent : String) {
contentProp.value = newContent
}
}
// after WebEngine loads successfully
val bridgeObj : BridgeObject = BridgeObject()
(webEngine.executeScript("window") as JSObject).setMember("bridgeObj", bridgeObj)
I am then calling these methods from Javascript, in TinyMCE setup:
ed.on('init', function() {
ed.setContent(initContent)
window.bridgeObj.setEditorAndSelection(ed)
window.bridgeObj.setInterfaceContent(ed.getContent())
});
ed.on('Change SelectionChange', function(e) {
window.bridgeObj.setInterfaceContent(ed.getContent())
});
Problem:
This works perfectly fine when the TinyMCEInterface class file lies in the root directory of my application (in package com.myself.app). That is, when the file structure looks like this:
com.myself.app/
│
├─ TinyMCEInterface.kt
│
├─ Main.kt
But breaks when I move TinyMCEInterface into a package (in package com.myself.app.somepackage):
com.myself.app/
│
├─ somepackage/
│ ├─ TinyMCEInterface.kt
│
├─ Main.kt
When I say "breaks", there are no errors; the calls to member functions of window.bridgeObj simply do not happen and quietly fail. I am completely bewildered as to how this can be happening.
Thanks in advance for any advice!

Fragment leaking in ViewPager2 with FragmentStateAdapter on onDestroyView()

FYI: The question is short, but just in case I added more information at the end that may be relevant.
I needed an infinite scrolling ViewPager2, and I wanted to reuse a Fragment from the project as it was already designed and calls already well stablished with its viewLifeCycle..
Also I am aware the VP recycles offscreen Fragments (1 offset position from Fragment shown) and has at least up to 3 Fragments at any given moment, so using Fragments was the choice.
The problem is that when going to the fourth page, the ViewPager2 tries to remove the first Fragment (as expected) and LeakCanary shows me this (Entire diagnosis at the end.):
D/LeakCanary: Watching instance of androidx.core.widget.NestedScrollView (com.****.****.ui.***.pages.add_element.SearchPageFragment2 received Fragment#onDestroyView() callback (references to its views should be cleared to prevent leaks)) with key 294cd9eb-3d6a-4c98-a69c-5d20e4c1652f
The diagnosis never points to my references, only to android library references.
Before the code bellow I had more lines, But I've been trimming them until kept with the bare minimum and the leak is still there.
// ----- onViewCreated() ------
MyPagerAdapter mPa = new MyPagerAdapter(
getChildFragmentManager(),
getViewLifecycleOwner().getLifecycle()
);
vp.setAdapter(mPa);
MyPagerAdapter.class that extends FragmentStateAdapter:
#NonNull
#Override
public Fragment createFragment(int position) {
return new SearchPageFragment2(); //Test Fragment
}
#Override
public int getItemCount() {
return 8; //Test fixed number
}
The Leaking Fragment:
public class SearchPageFragment2 extends Fragment {
#Nullable
#Override
public View onCreateView(#NonNull LayoutInflater inflater, #Nullable ViewGroup container, #Nullable Bundle savedInstanceState) {
return FragmentSearchPageBinding.inflate(inflater).getRoot();
}
}
What is causing the memory leak??
The question ENDS HERE.
Preamble...
The main View (the farthest Fragment ancestor at display while the leak is happening) is a BackStackEntry that we navigate to, from the Home Fragment, this View holds a toolBar with main info about the app, bellow the toolBar is the main content of this view, a ViewPager2 of fixed size with 3 Fragments, and on the first Fragment... a "MutableFrameLayout" that I created:
private final MutableFrameLayoutAdapter<ElementDBLoaderViewModel.FrameActions> adapter = new MutableFrameLayoutAdapter<>(
this, //Fragment owner
this::getChildFragmentManager, //FragmentManager supplier
() -> ElementDBLoaderViewModel.FrameActions.initiating, //initialValue
action -> { //Function<X, Fragment>
switch (action) {
case crossed:
return new AddElementExpandedFragment();
case not_crossed:
return new AddElementFragment();
case explore:
return new MainDBPaginationFragment2();
}
return null;
}
);
So that it can be used like this:
binding.fragmentContainer.setAdapter(adapter);
adapter.changeContent(ElementDBLoaderViewModel.FrameActions.crossed)
The component is leak proof, with hours of testing on different situations.
The main "engine" of this component:
......
if (oldFragment != null) {
FragmentTransaction ft = stackFm.beginTransaction();
ft.remove(oldFragment);
addCommit(ft, newFragment);
}
Where "stackFm" is the result of having acquired the FragmentManager supplier in the constructor with childFragmentManager.get() ("this::getChildFragmentManager")
......
private void addCommit(FragmentTransaction ft, Fragment newFragment) {
fragmentCreated.get().fragmentCreated(newFragment); //stateless adapter interface reference
ft.add(getId(), newFragment);
ft.commit();
}
......
The idea was to have a component easy to use with nothing fancy and straight forward.
Basically the first page (Fragment) of the ViewPager2 of fixed size where this MutableFrameLayout is placed, can take the form of 3 different Fragments (depending on DB size).
The Leaking ViewPager2 is inside MainDBPaginationFragment2.class, BUT BEFORE arriving to the MainDBPaginationFragment2 Fragment, we MUST first go through the AddElementExpandedFragment.class.
LEAK DIAGNOSIS (none of the references are mine)
┬───
│ GC Root: System class
│
├─ android.view.WindowManagerGlobal class
│ Leaking: NO (DecorView↓ is not leaking and a class is never leaking)
│ ↓ static WindowManagerGlobal.sDefaultWindowManager
├─ android.view.WindowManagerGlobal instance
│ Leaking: NO (DecorView↓ is not leaking)
│ ↓ WindowManagerGlobal.mViews
├─ java.util.ArrayList instance
│ Leaking: NO (DecorView↓ is not leaking)
│ ↓ ArrayList.elementData
├─ java.lang.Object[] array
│ Leaking: NO (DecorView↓ is not leaking)
│ ↓ Object[].[0]
├─ com.android.internal.policy.DecorView instance
│ Leaking: NO (ViewPager2$RecyclerViewImpl↓ is not leaking and View attached)
│ View is part of a window view hierarchy
│ View.mAttachInfo is not null (view attached)
│ View.mWindowAttachCount = 1
│ mContext instance of com.android.internal.policy.DecorContext, wrapping
│ activity com.****.****.ui.MainActivity with mDestroyed = false
│ ↓ DecorView.mAttachInfo
├─ android.view.View$AttachInfo instance
│ Leaking: NO (ViewPager2$RecyclerViewImpl↓ is not leaking)
│ ↓ View$AttachInfo.mScrollContainers
├─ java.util.ArrayList instance
│ Leaking: NO (ViewPager2$RecyclerViewImpl↓ is not leaking)
│ ↓ ArrayList.elementData
├─ java.lang.Object[] array
│ Leaking: NO (ViewPager2$RecyclerViewImpl↓ is not leaking)
│ ↓ Object[].[2]
├─ androidx.viewpager2.widget.ViewPager2$RecyclerViewImpl instance
│ Leaking: NO (View attached)
│ View is part of a window view hierarchy
│ View.mAttachInfo is not null (view attached)
│ View.mID = R.id.null
│ View.mWindowAttachCount = 1
│ mContext instance of com.****.****.ui.MainActivity with
│ mDestroyed = false
│ ↓ ViewPager2$RecyclerViewImpl.mRecycler
│ ~~~
├─ androidx.recyclerview.widget.RecyclerView$Recycler instance
│ Leaking: UNKNOWN
│ Retaining 48453 bytes in 442 objects
│ ↓ RecyclerView$Recycler.mRecyclerPool
│ ~~~~~
├─ androidx.recyclerview.widget.RecyclerView$RecycledViewPool instance
│ Leaking: UNKNOWN
│ Retaining 46192 bytes in 424 objects
│ ↓ RecyclerView$RecycledViewPool.mScrap
│ ~~
├─ android.util.SparseArray instance
│ Leaking: UNKNOWN
│ Retaining 46176 bytes in 423 objects
│ ↓ SparseArray.mValues
│ ~~~
├─ java.lang.Object[] array
│ Leaking: UNKNOWN
│ Retaining 46111 bytes in 421 objects
│ ↓ Object[].[0]
│ ~
├─ androidx.recyclerview.widget.RecyclerView$RecycledViewPool$ScrapData instance
│ Leaking: UNKNOWN
│ Retaining 46067 bytes in 420 objects
│ ↓ RecyclerView$RecycledViewPool$ScrapData.mScrapHeap
│ ~~~~
├─ java.util.ArrayList instance
│ Leaking: UNKNOWN
│ Retaining 46035 bytes in 419 objects
│ ↓ ArrayList.elementData
│ ~~~~~
├─ java.lang.Object[] array
│ Leaking: UNKNOWN
│ Retaining 46015 bytes in 418 objects
│ ↓ Object[].[0]
│ ~
├─ androidx.viewpager2.adapter.FragmentViewHolder instance
│ Leaking: UNKNOWN
│ Retaining 43762 bytes in 400 objects
│ ↓ FragmentViewHolder.itemView
│ ~~~~
├─ android.widget.FrameLayout instance
│ Leaking: UNKNOWN
│ Retaining 43677 bytes in 399 objects
│ View not part of a window view hierarchy
│ View.mAttachInfo is null (view detached)
│ View.mID = R.id.null
│ View.mWindowAttachCount = 1
│ mContext instance of com.****.****.ui.MainActivity with
│ mDestroyed = false
│ ↓ FrameLayout.mMatchParentChildren
│ ~~~~~~~~
├─ java.util.ArrayList instance
│ Leaking: UNKNOWN
│ Retaining 41573 bytes in 385 objects
│ ↓ ArrayList.elementData
│ ~~~~~
├─ java.lang.Object[] array
│ Leaking: UNKNOWN
│ Retaining 41553 bytes in 384 objects
│ ↓ Object[].[0]
│ ~
╰→ androidx.core.widget.NestedScrollView instance
​ Leaking: YES (ObjectWatcher was watching this because com.****.
​ ****.ui.****.pages.add_element.SearchPageFragment2
​ received Fragment#onDestroyView() callback (references to its views
​ should be cleared to prevent leaks))
​ Retaining 41549 bytes in 383 objects
​ key = 294cd9eb-3d6a-4c98-a69c-5d20e4c1652f
​ watchDurationMillis = 7979
​ retainedDurationMillis = 2978
​ View not part of a window view hierarchy
​ View.mAttachInfo is null (view detached)
​ View.mID = R.id.scrolling_content_table
​ View.mWindowAttachCount = 1
​ mContext instance of com.****.****.ui.MainActivity with
​ mDestroyed = false
The solution for me at least was stop using the FragmentStateAdapter, and instead use a RecyclerViewAdapter, and submit to this adapter, a list of integers representing the index of each page (needed for db live connection), while at the same time determining the size of the batch of pages (given by the initial DB request).
Finally, on the view creation of each viewHolder, the content of each page was fed including resolved viewmodel contents.
A vital AND hardest part of the solution was granting each viewHolder its own lifeCycle, so that each could manage its db connection on its own.
As you can guess the code is horrendous, and even if I tried obsessively to order it the best I could, it is still not sufficiently organized IMO.
My interpretation of this is that doing pagination on your own is absolutely nightmarish... but possible.
Also encapsulating pagination functions such as page controls, db requests and display, make it so that the entire "concept" of pagination itself becomes a single unit on its own, making it absolutely difficult to separate each component on its own capsule as the function of every component is deeply interconnected with its consumer.
One example is the way your DB queries assume "pivot points" (whether they are inclusive or not, or maybe they are not even required at all and you can randomly jump at any page(index) you want), I am not sufficiently familiar with other db but if this changes enough across services, one may even need to change how the entire pagination works, bottom - up.
The components that could definitely be encapsulated are: the adapter, the "scroller" of your data, that will define the number of pages and page position while defining page control functions AND a viewmodel that will simply serve as an envelope to the scroller itself, but that's it. the actual display needs to be interwoven with all these components in the most savage way possible. and your scroller must have multiple sources of input/output (db input: local and remote, user input (page controller), graphical outputs: pages and page controller)

Meteor.setInterval breaking after +1 day of uptime (callback: undefined)

we are experiencing Exception in setInterval callback: undefined for intervals after a few days of them running on an hourly basis. Restarting resolves it for a few days until it breaks like this again. What could be the cause of this?
(there is nothing else being logged/thrown, no stack trace either)
Meteor.setInterval(
startJob(),
1000 * 60 * 60
);
Meteor: 1.8.0.2
The most likely cause of the crash is that you are calling setInterval repeatedly. In the short term it's ok, but you end up with many interval timers running, and that is eventually going to be a problem because of the memory and processing load.
I prefer to use the npm package node-cron, I set up the timer in a startup method like this:
//
// These are cron-style time specifiers
//
// ┌───────────── minute (0 - 59)
// │ ┌───────────── hour (0 - 23)
// │ │ ┌───────────── day of month (1 - 31)
// │ │ │ ┌───────────── month (1 - 12)
// │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)
// │ │ │ │ │ 7 is also Sunday on some systems)
// │ │ │ │ │
// │ │ │ │ │
// * * * * *
const TICKER_INTERVAL = '1,16,31,46 * * * *' // This runs 4 times an hour
// const TICKER_INTERVAL = '* * * * *' // This runs every minute (use when debugging)
Meteor.startup(() => {
cron.schedule(TICKER_INTERVAL, Meteor.bindEnvironment(signoutTicker))
})
This only applies on the server of course (you knew that, didn't you?)
I find this works very reliably for me.

.NET Angular using RequireJS

I created a new ASP.NET Empty Web Application in Visual Studios 2013. I have an angular project with the following directory structure:
├───TestProject
│ ├───index.html
│ ├───package.json
│ ├───Web.config
│ ├───app
│ │ ├───app.module.js
│ │ ├───main.js
│ ├───bin
│ ├───node_modules
│ │ ├───google-distance
│ │ │ └───test
│ │ ├───json-stringify-safe
│ │ │ └───test
│ │ ├───qs
│ │ ├───request
│ │ │ └───lib
│ ├───obj
│ │ └───Debug
│ │ └───TempPE
│ ├───Properties
│ └───Scripts
└───packages
├───angularjs.1.6.1
│ └───content
│ └───Scripts
│ └───i18n
│ └───ngLocale
├───AngularJS.Core.1.6.1
│ └───content
│ └───Scripts
└───RequireJS.2.3.2
└───content
└───Scripts
Here's what my packages.json looks like:
{
"name": "google-distance",
"version": "1.0.1",
"main": "index",
"description": "A simple node.js wrapper for Google's Distance Matrix API",
"author": {
"name": "Edward Look",
"email": "edwlook#gmail.com",
"url": "http://edwardlook.com"
},
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/edwlook/node-google-distance.git"
},
"keywords": [
"google",
"maps",
"distance",
"matrix",
"api"
],
"dependencies": {
"google-distance": "~1.0.1"
}
}
I ran npm install and it created the node_modules folder. At the top of my main.js file I have:
var distance = require('../node_modules/google-distance/index.js');
But every time I load the page the console says:
require.js:168 Uncaught Error: Module name "../node_modules/google-distance/index.js" has not been loaded yet for context: _. Use require([])
http://requirejs.org/docs/errors.html#notloaded
at makeError (require.js:168)
at Object.localRequire [as require] (require.js:1433)
at requirejs (require.js:1794)
at main.js:2
What am I doing wrong? How do I use the google-distance module?
Normally you should require it as follows;
var distance = require('google-distance');
The way node will look for modules is as follows;
It will perform a hierarchical directory search for "node_modules" and "google-distance" in the following ways:
./node_modules/google-distance.js
./node_modules/google-distance/index.js
./node_modules/google-distance/package.json
Also, it looks like you should use the asynchronous callback version of require to load this library.
require(['google-distance'], function (distance) {
//package is now loaded.
});
From the require FAQ:
MODULE NAME ... HAS NOT BEEN LOADED YET FOR CONTEXT: ... This occurs
when there is a require('name') call, but the 'name' module has not
been loaded yet. If the error message includes Use require([]), then
it was a top-level require call (not a require call inside a define()
call) that should be using the async, callback version of require to
load the code.
http://requirejs.org/docs/errors.html#notloaded

Resources