I wondered if there is anything available to scrape my linkedin network connections. For example if I have 1 connection on linkedIn I would like to get my connection's connections, so if my connection has 200 connections I would like to get that data if the profile is set to public.
I've tried linkedin helper extension but it's only scraping my connections, without further drilling into my networks connections.
I've tried running scrapedin npm module but it gives me errors on the implementation.
const scrapedin = require('scrapedin')
const profileScraper = await scrapedin({ email: 'login#mail.com', password: 'pass' })
const profile = await profileScraper('https://www.linkedin.com/in/some-profile/')
Expected result would be an csv with my networks connections which will be a big list if my connections have a lot of other connections. Does anyone used any crawlers or implemented this with the new linkedin website?
Are you looking for this? It implements scrapedIn to aggregate connections.
https://github.com/linkedtales/scrapedin-linkedin-crawler
Related
I'm writing some R code that queries a MongoDB database, imports records matching the query criteria into R, performs record linkage with another data source, and then pushes the updated records back into MongoDB.
The code needs to work with any instance of the MongoDB database. Some people have it installed as standalone on their own computers, while others have it installed on their organisational servers. Note that these are servers specific to individual organisations and not the public mongo server.
To test my code, I have access to both scenarios - one instance is set up on my own computer, and I have several remote server instances as well.
The MongoDB database has some APIs, but I was struggling with the adapting the APIs to include the correct syntax to form my query, so I thought I would try the mongolite package instead.
I was able to create a successful connection string for the MongoDB instance on my local computer, using my user ID (which I retrieve first with an API and save as the R object myids), password, the localhost and port number as below:
# Load library:
library(mongolite)
# Create connection:
con <- mongolite::mongo(collection = "person",
db = "go-data",
url = paste0("mongodb://localhost:3000",
myids$userid,
":",
rawToChar(password)))
I understood from reading the mongolite user manual that to create the connection string / URI, you skip the http or https part of the address and preface it with either mongodb:// when the Mongodb database is on a local computer, or mongodb+srv:// when the Mongodb database is on a remote server.
However, when I try just changing the prefix and login details for the remote server version, the connection fails. Say the URL for my remote server is https://mydb-r21.orgname.org/ which opens a web page where you can log in to the Mongodb database and interact with it via a graphical user interface. Just swapping localhost:3000 for the web address mydb-r21.orgname.org/ and supplying the relevant login credentials for that server doesn't work:
# Load library:
library(mongolite)
# Create connection:
con <- mongolite::mongo(collection = "person",
db = "go-data",
url = paste0("mongodb+srv://mydb-r21.orgname.org/",
myids$userid,
":",
rawToChar(password)))
When I try, this is the error I get:
Warning: [ERROR] Failed to look up SRV record "_mongodb._tcp.mydb-r21.orgname.org": DNS name does not exist.
Error: Invalid uri_string. Try mongodb://localhost
If I try changing to mongodb::// (not localhost because it isn't hosted locally) I get this:
Error: No suitable servers found (`serverSelectionTryOnce` set): [connection timeout calling hello on 'mydb-r21.orgname.org:27017']
Interestingly, the port that is suffixed in the error message is the correct one that I was expecting, but that still doesn't help me.
The documentation in the mongolite user manual and other places I've found online seem to add some read/write specifications to the connection string, but as I'm not very familiar with how connection strings are constructed, I don't know if these are specific to the databases they are using in their examples. I can't find any clear explanation of what the extra bits that are not part of the URL mean, e.g. as shown in this blog. All the prefixes seem to be a bit different too, so I am not even sure what would be appropriate to try in my case.
Can anyone explain why the connection string works fine with localhost:port number for the local instance, but doesn't work with the URL for the remote server / online instance?
Also what do I need to do to make the URI for the remote server valid?
I am developing an OPC UA client that needs to connect asynchronously to a server and update its node data. I am using UA_Client_connectAsync to connect to the server but the async write/read APIs e.g. UA_Client_writeValueAttribute_async in
Examples/client_async.c
of Examples codes are missing from source code and do not compile. Are they deprecated? What is the new API for Async read/write to the server?
I am asking this question because I have tried the UA_Client_writeValueAttribute API with the non-blocking UA_Client_connectAsync and the server node does not receive any data on its node.
I am not using real-time asterisk , But still astdb.sqlite3 contains entries of online peers with Reg.Contact information in SIP/registry/peer. key . I would like to store contact information of all peers as they come online in a separate persistent database. I need this for sending push notifications by fetching deviceID etc information in registration contact .
I tried to pull this information from astdb.sqlite3 but the entries are clearing off as soon as devices go offline .Though I am able to fetch the information with "sip show peer XXXX" in asterisk CLI , It is overburdened to fetch every time like this . Instead I want to save only Regcontact information for all the devices in a database ( without realtime) as the devices come online. The other way I tried to pull the information is using AMI event listener. But with AMI I don't see complete information like contact information It displays only below information
Event: PeerStatus
Privilege: system,all
SequenceNumber: 75
File: manager.c
Line: 1856
Func: manager_default_msg_cb
ChannelType: SIP
Peer: SIP/2030
PeerStatus: Reachable
Can someone suggest a better way to push Only Regcontact information to a database as the devices come online .
There are no mechanism like that in asterisk.
You can use kamailio or write patch similar to this one https://reviewboard.asterisk.org/r/4490/
It sounds like you have dynamic IPs for your endpoints, and you want a way to update a separate DB as soon as a device registers with an IP/port pair.
If you enable the security log, you will see all auth events, including the "SuccessfulAuth" event, which includes the RemoteAddress of the endpoint (including port and protocol).
Here is an example line:
[Jul 21 19:53:45] SECURITY[1342] res_security_log.c: SecurityEvent="SuccessfulAuth",EventTV="2020-07-21T19:53:45.182+0000",Severity="Informational",Service="SIP",EventVersion="1",AccountID="102",SessionID="0x7f41040132c0",LocalAddress="IPV4/UDP/10.0.0.200/5060",RemoteAddress="IPV4/UDP/10.0.0.75/5062",UsingPassword="1"
If all you're looking for is AccountID="102" and RemoteAddress="IPV4/UDP/10.0.0.75/5062", a very fast/cheap way to get it is to enable the security log, and use a script to tail it and update your DB as soon as the event occurs. I like to keep the security log on anyways for utilities like fail2ban. Just make sure your script is able to reopen the file each time it is rotated.
Edit:
By default the log is in /var/log/asterisk. To enable it, edit /etc/asterisk/logger.conf and un-comment (or create) the line under [logfiles] that says security => security.
Firebase:
{exampleValue: "something";}
Javascript:
var ref = new Firebase(url + "/exampleValue");
ref.set("something");
I have a couple of questions around setting a value in Firebase to the same value, as shown above.
Does it use bandwidth for the user?
Does it count towards the bandwidth for my Firebase plan?
tl;dr: calling set() multiple times with the same value will use bandwidth each time.
You can easily test these things yourself. I just did the following: open the "Network" tab in the Chrome developer tools and find the WebSocket ("WS") that communicates with Firebase. In the "Frame" tab of that web socket you can see exactly what the Firebase client communicates with its server.
Then run the following in the developer console:
var newRef = ref.push();
This generates a new path/location, but does not send anything to the Firebase servers yet. Then:
newRef.set('This is a value');
This sets the value, so uses bandwidth.
Then I ran the same again:
newRef.set('This is a value');
This once again sent data to the Firebase server, so it uses bandwidth.
When you think about it, this also makes sense. The Firebase client knows only its local state. The value on the server might be different, for example because somebody else has changed the value. So when you tell it to set the value, it must send that command to the server to ensure that value actually gets written into the database.
Is there any way to write a security rule or is there any other approach that would make possible only for currently connected (not authenticated) user to write/read certain location - admin should also be able to write/read?
Can a rule be written that disallows users to read of complete list of entries and let them read only entry that matches some identifier that was passed from client?
I'm trying to exchange some data between user and Node.js application through Firebase and that data shouldn't be able to read or write by anyone else other than user and/or admin.
I know that one solution would be that user requests auth token on my server and uses it to authenticate on Firebase and that would make it possible to write rule which prevents reads and writes. However, I'm trying to avoid user connecting to my server so this solution is not first option.
This is in a way session based scenario which is not available in Firebase but I have
some ideas that could solve this kind of problem - if implemented before session management:
maybe letting admin write into /.info/ location which is observed by client for every change and can be read only by active connection - if I understood correctly how .info works
maybe creating .temp location for that purpose
maybe letting admin and connected client could have more access to connection information which would contain some connection unique id, that can be used to create location with that name and use it inside rule to prevent reading and listing to other users
Thanks
This seems like a classic XY problem (i.e. trying to solve the attempted solution instead of the actual problem).
If I understand your constraints correctly, the underlying issue is that you do not wish to have direct connections to your server. This is currently the model we're using with Firebase and I can think of two simple patterns to accomplish this.
1) Store the data in an non-guessable path
Create a UUID or GID or, assuming we're not talking bank level security here, just a plain Firebase ID ( firebaseRef.push().name() ). Then have the server and client communicate via this path.
This avoids the need for security rules since the URLs are unguessable, or close enough to it, in the case of the Firebase ID, for normal uses.
Client example:
var fb = new Firebase(MY_INSTANCE_URL+'/connect');
var uniquePath = fb.push();
var myId = uniquePath.name();
// send a message to the server
uniquePath.push('hello world');
From the server, simply monitor connect, each one that connects is a new client:
var fb = new Firebase(MY_INSTANCE_URL+'/connect');
fb.on('child_added', newClientConnected);
function newClientConnected(snapshot) {
snapshot.ref().on('child_added', function(ss) {
// when the client sends me a message, log it and then return "goodbye"
console.log('new message', ss.val());
ss.ref().set('goodbye');
});
};
In your security rules:
{
"rules": {
// read/write are false by default
"connect": {
// contents cannot be listed, no way to find out ids other than guessing
"$client": {
".read": true,
".write": true
}
}
}
}
2) Use Firebase authentication
Instead of expending so much effort to avoid authentication, just use a third party service, like Firebase's built-in auth, or Singly (which supports Firebase). This is the best of both worlds, and the model I use for most cases.
Your client can authenticate directly with one of these services, never touching your server, and then authenticate to Firebase with the token, allowing security rules to take effect.