Adding a connector to Kafka Connect - jar

I am using Confluent Kafka Docker image, specifically using this: https://github.com/confluentinc/cp-docker-images/tree/4.0.x/examples/cp-all-in-one
I want to add the MySQL connector, by:
downloading the version 1.5.46 of the connector (https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz)
mounting a volume with a jar that comes out of the downloaded connector (mysql-connector-java-5.1.46-bin.jar)
adding the CONNECT_PLUGIN_PATH to the docker compose file
Basically, extrapolating the part of interest form the compose file, this is the important snippet:
environment:
<cut>
CONNECT_PLUGIN_PATH: '/etc/kafka-connect/jars/'
volumes:
- "./mysql_connector:/etc/kafka-connect/jars/"
The JAR is correctly present in the Docker container.
But, of course, if I try to use the connector, I can't find in the list of the enabled connectors. If I go to control center, I see:
[{
"class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
"type": "sink",
"version": "1.0.0-cp1"
}, {
"class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
"type": "source",
"version": "1.0.0-cp1"
}]

Your plugin path needs to contain the path to the actual JDBC Connect JAR files as well as the MySQL JAR
If you mount the MySQL JAR under this path
/usr/share/java/kafka-connect-jdbc
Then set the CONNECT_PLUGIN_PATH=/usr/share/java, then you have access to all bundled connectors. And the JDBC Connector will be able to read/write MySQL data.

Related

Cloud Build Trigger - Not in Firebase app directory error

I have problem with deploying node.js (Angular) application from Bitbucket repository to Firebase using Cloud Build Trigger.
I performed steps from this article - https://cloud.google.com/build/docs/deploying-builds/deploy-firebase
Necessary APIs are turned on,
Service account has all required permissions,
Firebase community builder is deployed and visible in Container Registry,
Cloudbuild.json file is added to repository,
Cloud Build Trigger is created and it points to one specific branch of my repository.
Problem is that after running Cloud Build Trigger I receive following error: "Error: Not in Firebase app directory (could not locate firebase.json)"
What could be reason of such error? Is it possible to point to the trigger where in repository is firebase application and firebase.json file?
EDIT:
My cloudbuild.json file is quite simple:
{
"steps": [
{
"name": "gcr.io/PROJECT_NAME/firebase",
"args": [
"deploy",
"--project",
"PROJECT_NAME",
"--only",
"functions"
]
}
]
}
Logs in Cloud Build Trigger history:
starting build "build_id"
FETCHSOURCE
Initialized empty Git repository in /workspace/.git/
From https://source.developers.google.com/p/link
* branch branch_id -> FETCH_HEAD
HEAD is now at d8bc2f1 Add cloudbuild.json file
BUILD
Pulling image: gcr.io/PROJECT/firebase
Using default tag: latest
latest: Pulling PROJECT/firebase
1e987daa2432: Already exists
a0edb687a3da: Already exists
6891892cc2ec: Already exists
684eb726ddc5: Already exists
b0af097f0da6: Already exists
154aee36a7da: Already exists
37e5835696f7: Pulling fs layer
62eb6e670f1d: Pulling fs layer
47e62615d9f9: Pulling fs layer
428cea824ccd: Pulling fs layer
765a2c722bf9: Pulling fs layer
b3f5d0a285e3: Pulling fs layer
428cea824ccd: Waiting
765a2c722bf9: Waiting
b3f5d0a285e3: Waiting
47e62615d9f9: Verifying Checksum
47e62615d9f9: Download complete
62eb6e670f1d: Verifying Checksum
62eb6e670f1d: Download complete
765a2c722bf9: Verifying Checksum
765a2c722bf9: Download complete
b3f5d0a285e3: Verifying Checksum
b3f5d0a285e3: Download complete
37e5835696f7: Verifying Checksum
37e5835696f7: Download complete
428cea824ccd: Verifying Checksum
428cea824ccd: Download complete
37e5835696f7: Pull complete
62eb6e670f1d: Pull complete
47e62615d9f9: Pull complete
428cea824ccd: Pull complete
765a2c722bf9: Pull complete
b3f5d0a285e3: Pull complete
Digest: sha256:4b6b7214d6344c8247130bf3f5443a2851e39aed3ececb32dfb2cc25a5c07e44
Status: Downloaded newer image for gcr.io/PROJECT/firebase:latest
gcr.io/PROJECT/firebase:latest
Error: Not in a Firebase app directory (could not locate firebase.json)
ERROR
ERROR: build step 0 "gcr.io/PROJECT/firebase" failed: step exited with non-zero status: 1
Firebase.json file is placed in repository in following path: /apps/firebase/firebase.json
I can't see any possibility to point that path in Cloud Build Trigger config.
I'm pretty sure that you aren't in the correct directory when you run the command. To check this, you can add this step
{
"steps": [
{
"name": "gcr.io/cloud-builders/gcloud",
"entrypoint":"ls"
"args": ["-la"]
}
]
}
If you aren't in the correct directory, and you need to be in the ./app/firebase to run your deploy, you can add this parameter in your step
{
"steps": [
{
"name": "gcr.io/PROJECT_NAME/firebase",
"dir": "app/firebase",
"args": [
"deploy",
"--project",
"PROJECT_NAME",
"--only",
"functions"
]
}
]
}
Let me know if it's better

Unable to read from Jfrog Artifactory

I am attempting to connect to Artifactory to read a Java zip file with the admin user. The call is made through Talend Restful API interface (described below).
Expected result:
To read the file stored in Artifact repository and copy it to a server
To see a record of the access in the access.log file
Actual:
There is a line in the request.log with the HTTP code 200 , but there is nothing recorded in the access.log. How do I find out what the issue is?
About Talend Restful API:
Talend has a way to call its Java based jobs through its Talend Administrator Console (TAC), a web based interface by creating an object called a task. The task refers to a Java zipped file stored in Artifactory, connects to Artifactory and copies it to the machine where TAC runs and executes it. When this action is done through TAC itself, it works. But when it is called through Talend Restful API, we get the error "{"error":"nexus.error.wrongstatus","returnCode":181}"
The Restful call (calling from TAC server):
./MetaServletCaller.sh --tac-url http://... -json '{"actionName": "associatePreGeneratedJob", "active": true, "artifactoryArtifactId": "job_showMsg2", "artifactoryGroupId": "test.job", "artifactoryRepository": "snapshot", "artifactoryVersion": "0.1.0", "authPass": "xxx", "authUser": "xxx", "contextName": "Default", "description": "test_jb1 description", "executionServerName": "jbsrv", "importType": "Artifactory", "logLevel": "Info", "onUnknownStateJob": "WAIT", "pauseOnError": false, "taskName": "job_test", "taskType": "Artifact","targetConductor":"JOBCONDUCTOR", "timeout": 3600}'

Packer: Sample file for Openstack

Can someone please provide me with a sample file for Packer creating OS image? I have this one:
{
"builders": [{
"type": "openstack",
"ssh_username": "ubuntu",
"tenant_name": "mytenant",
"flavor": "m1.tiny",
"identity_endpoint": "http://1.2.3.4:5000/",
"availability_zone": "az1",
"source_image": "Ubuntu 16.04 With Proxy",
"image_name": "Ubuntu 16.04 With Proxy and Python"
}],
"provisioners": [
{
"type": "shell",
"script": "python.sh"
}
]
}
but OS always returns:
==> openstack: Error launching source server: Invalid request due to incorrect syntax or missing required parameters.
I have no idea what I am missing.
Of course i have correct OS_ env values preset for my Nova API.
You have to use source_image_name or use the ID to reference the image
From the docs:
source_image (string) - The ID or full URL to the base image to use. This is the image that will be used to launch a new server and provision it. Unless you specify completely custom SSH settings, the source image must have cloud-init installed so that the keypair gets assigned properly.
source_image_name (string) - The name of the base image to use. This is an alternative way of providing source_image and only either of them can be specified.
See source_image

retrieve artifact with maven timestamp from artifactory

Is there a way to retrieve an artifact with the maven timestamp as it was originally uploaded by maven?
from jenkins logs:
Uploading: http://artifactory.foo/artifactory/libs-snapshot-local/com/foo/foo-web-service/1.16.0-SNAPSHOT/foo-web-service-1.16.0-20160504.182015-2.tar.gz
Results from artifactory REST api:
$ curl -X GET 'http://artifactory.foo/artifactory/api/search/gavc?g=com.foo&a=foo-web-service&v=1.16.0-SNAPSHOT&c=*&repos=libs-snapshot-local'
{
"results" : [ {
"uri" : "http://artifactory.foo/artifactory/api/storage/libs-snapshot-local/com/foo/foo-web-service/1.16.0-SNAPSHOT/foo-web-service-1.16.0-SNAPSHOT-sources.jar"
}, {
"uri" : "http://artifactory.foo/artifactory/api/storage/libs-snapshot-local/com/foo/foo-web-service/1.16.0-SNAPSHOT/foo-web-service-1.16.0-SNAPSHOT.pom"
}, {
"uri" : "http://artifactory.foo/artifactory/api/storage/libs-snapshot-local/com/foo/foo-web-service/1.16.0-SNAPSHOT/foo-web-service-1.16.0-SNAPSHOT.tar.gz"
}, {
"uri" : "http://artifactory.foo/artifactory/api/storage/libs-snapshot-local/com/foo/foo-web-service/1.16.0-SNAPSHOT/foo-web-service-1.16.0-SNAPSHOT.war"
} ]
}
I'd like to get the the same name as it was uploaded to via a wget or equivalent...
What I want to acheive:
jenkins uploads foo-web-service-1.16.0-20160504.182015-2.tar.gz to libs-snapshot-local
query REST api to get latest artifact link that includes the timestamps in the name with parameters a=foo-web-service&version=1.16.0&...
wget $artifact_link_with_timestamp
What I currently acheive that does not satisfy my need:
jenkins uploads foo-web-service-1.16.0-20160504.182015-2.tar.gz to libs-snapshot-local
query REST api via gavc search with parameters a=foo-web-service&version=1.16.0&...
wget $artifact_link
Conclusion as stated in the accepted answer, the problem was in the artifactory config itself. To achieve what I wanted, I needed the snapshots to be unique.
As long as your repository is configured to use unique snapshots (or to use client snapshot policy and you use Maven 3 and up), you can always use the Maven timestamp as a version. Replacing it with -SNAPSHOT is a "runtime" trick to make the resolution easier.
If your repository is configured to use non-unique snapshots, the files are actually stored with -SNAPSHOT instead of version and override previous snapshots (don't do that).

Deploying Telescope App to Amazon EC2

What am I doing wrong? I'm trying to deploy my Telescope app. I'm using this to do that: https://github.com/arunoda/meteor-up. It gives me an error when I try to deploy using Mup. Here's my mup.json config file:
{
// Server authentication info
"servers": [
{
"host": "52.25.228.14",
"username": "ec2-user",
//"password": "password"
// or pem file (ssh based authentication)
"pem": "~/Documents/appname/appname.pem"
}
],
// Install MongoDB in the server, does not destroy local MongoDB on future setup
"setupMongo": true,
// WARNING: Node.js is required! Only skip if you already have Node.js installed on server.
"setupNode": true,
// WARNING: If nodeVersion omitted will setup 0.10.36 by default. Do not use v, only version number.
"nodeVersion": "0.10.36",
// Install PhantomJS in the server
"setupPhantom": true,
// Show a progress bar during the upload of the bundle to the server.
// Might cause an error in some rare cases if set to true, for instance in Shippable CI
"enableUploadProgressBar": true,
// Application name (No spaces)
"appName": "appname",
// Location of app (local directory)
"app": "~/Documents/appname/Telescope",
// Configure environment
"env": {
"ROOT_URL": "ec2-52-25-228-14.us-west-2.compute.amazonaws.com"
},
// Meteor Up checks if the app comes online just after the deployment
// before mup checks that, it will wait for no. of seconds configured below
"deployCheckWaitTime": 15
}
Common pitfalls :
Are sure you ran mup setup before deploying ?
Try using a relative path to setup the app location, ie replace ~/Documents/appname/Telescope by . if your mup.json is located at the root of your Telescope instance.
I did run mup setup, did use http://, and tried many times to get it to connect/deploy.
I gave up and just went with a Wordpress theme and am customizing it how I want it. The other way was way too complicated. It didn't allow me to do what I wanted in a timely manner. Thanks for trying to help though guys.

Resources