I have found similar posts but they are rather old and they sadly did not solve my issue.
I am trying to add a Gateway for my GoLang GRPC services
PROJ_PATH=${CURDIR}
.PHONY: proto
proto: ## Generate protobuf code
# Compile proto files inside the project.
protoc api.proto --proto_path=${PROJ_PATH}/proto --go_out=. --go-grpc_out=. \
--grpc-gateway_out . \
--grpc-gateway_opt generate_unbound_methods=true \
--openapiv2_out . \
--openapiv2_opt logtostderr=true \
--openapiv2_opt generate_unbound_methods=true
I add the two dependencies
go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2
But I still get his error
protoc-gen-grpc-gateway: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--grpc-gateway_out: protoc-gen-grpc-gateway: Plugin failed with status code 1.
make: *** [proto] Error 1
I had to remove the import and gateway options in the GRPC file because this error so it looks like I have the wrong plugin but reading here tells me otherwise.
google/api/annotations.proto: File not found.
api.proto:5:1: Import "google/api/annotations.proto" was not found or had errors.
make: *** [proto] Error 1
syntax = "proto3";
option go_package = "pkg/api";
import "google/api/annotations.proto"; <---- had to remove
service ApiService {
rpc Test(TestRequest) returns (TestResponse){
option (google.api.http) = { <---- had to remove
get: "/v1/test"
body: "*"
};
}
}
Any advice would be greatly appreciated.
---- solved with ------
go install \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway#latest \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2#latest
then add the path to the proto command
--plugin=protoc-gen-grpc-gateway=${GOPATH}/bin/protoc-gen-grpc-gateway \
full command...
protoc api.proto --proto_path=${PROJ_PATH}/proto --go_out=. --go-grpc_out=. \
--grpc-gateway_out . \
--grpc-gateway_opt generate_unbound_methods=true \
--plugin=protoc-gen-grpc-gateway=${GOPATH}/bin/protoc-gen-grpc-gateway \
--openapiv2_out . \
--openapiv2_opt logtostderr=true \
--openapiv2_opt generate_unbound_methods=true
I had to install the binary like so
go install \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway#latest \
github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2#latest
Then update the proto command with the path.
--plugin=protoc-gen-grpc-gateway=${GOPATH}/bin/protoc-gen-grpc-gateway \
Full proto command
PROJ_PATH=${CURDIR}
.PHONY: proto
proto: ## Generate protobuf code
# Compile proto files inside the project.
protoc api.proto --proto_path=${PROJ_PATH}/proto --go_out=. --go-grpc_out=. \
--grpc-gateway_out . \
--grpc-gateway_opt generate_unbound_methods=true \
--plugin=protoc-gen-grpc-gateway=${GOPATH}/bin/protoc-gen-grpc-gateway \
--openapiv2_out . \
--openapiv2_opt logtostderr=true \
--openapiv2_opt generate_unbound_methods=true
I am new to setting up the gRPC web based client side. Our backend is already up and running on Go with gRPC. I am testing out what it's like converting the .proto file into TS. I am successfully able to generate some of the files, however, I am missing the TypeScript "Service" file.
I pretty much followed the instructions from the grpc_tools_node_protoc_ts site.
Setup a script to generate files for 1) the service and 2) the client model:
PROTOC_GEN_TS_PATH="./node_modules/.bin/protoc-gen-ts"
GRPC_TOOLS_NODE_PROTOC_PLUGIN="./node_modules/.bin/grpc_tools_node_protoc_plugin"
GRPC_TOOLS_NODE_PROTOC="./node_modules/.bin/grpc_tools_node_protoc"
OUT_DIR="./_protos_/proto/"
# JavaScript code generating
${GRPC_TOOLS_NODE_PROTOC} \
--plugin=protoc-gen-grpc="${GRPC_TOOLS_NODE_PROTOC_PLUGIN}" \
--js_out=import_style=commonjs,binary:"${OUT_DIR}" \
--grpc_out="${OUT_DIR}" \
-I "${OUT_DIR}" \
"${OUT_DIR}"/*.proto
${GRPC_TOOLS_NODE_PROTOC} \
--plugin=protoc-gen-ts="${PROTOC_GEN_TS_PATH}" \
--ts_out="${OUT_DIR}" \
-I "${OUT_DIR}" \
"${OUT_DIR}"/*.proto
What I get on the output is missing the *_grpc_pb.d.ts. I am under the impression I need this? 🤷🏻♂️
I have also tried adding the service option to the flag:
--ts_out="service=grpc-web:${OUT_DIR}" \
This now generates a *_pb_service.d.ts output file, still without the *_grpc_pb.d.ts file. I was reading the docs more and am thinking this service=grpc-web is actually the option I need since we're not running a node server.
Does this seem right? This is what I have now:
# Note the ts_out flag "service=grpc-node":
# This does generate the *_grpc_pb.d.ts but not the service files
protoc \
--plugin="protoc-gen-ts=${PROTOC_GEN_TS_PATH}" \
--plugin=protoc-gen-grpc=${GRPC_TOOLS_NODE_PROTOC_PLUGIN} \
--js_out="import_style=commonjs,binary:${OUT_DIR}" \
--ts_out="service=grpc-node:${OUT_DIR}" \
--grpc_out="${OUT_DIR}" \
-I "${OUT_DIR}" \
"${OUT_DIR}"/*.proto
# Note the ts_out flag "service=grpc-web":
# This does generate the service files, but not the *_grpc_pb.d.ts file
protoc \
--plugin="protoc-gen-ts=${PROTOC_GEN_TS_PATH}" \
--plugin=protoc-gen-grpc=${GRPC_TOOLS_NODE_PROTOC_PLUGIN} \
--js_out="import_style=commonjs,binary:${OUT_DIR}" \
--ts_out="service=grpc-web:${OUT_DIR}" \
--grpc_out="${OUT_DIR}" \
-I "${OUT_DIR}" \
"${OUT_DIR}"/*.proto
I am running a simple map reduce job in Azure HDInsight,below is the command that we are running:
java -jar WordCount201.jar wasb://hexhadoopcluster-2019-05-15t07-01-07-193z#hexanikahdinsight.blob.core.windows.net/hexa/CustData.csv wasb://hexhadoopcluster-2019-05-15t07-01-07-193z#hexanikahdinsight.blob.core.windows.net/hexa
Getting the below error :
java.io.IOException: No FileSystem for scheme: wasb
For Java use jdk1.8 and below POM org.apache.hadoop hadoop-mapreduce-examples2.7.3scope>provided org.apache.hadoophadoop-mapreduce-client-common2.7.3providedjdk.toolsjdk.toolsorg.apache.hadoophadoop-common2.7.3provided
WASB is a wrapper around HDFS file system. I am not sure you can use it in normal java program. Do you have any reference / link which you referred to?
You can try to get the https equivalent of the custData.csv file.Below is an example of Spark job I am able to submit on HDInsight cluster using WASB
spark-submit \
--class com.nileshgule.movielens.MovieRatingAnalysis \
--master yarn \
--deploy-mode cluster \
--executor-memory 1g \
--name MoviesCsvReader \
--conf "spark.app.id=MovieRatingAnalysis" \
wasb://hd-spark-cluster-2019#hdsparkclusterstorage.blob.core.windows.net/learning-spark-1.0.jar \
wasb://hd-spark-cluster-2019#hdsparkclusterstorage.blob.core.windows.net/ml-latest/ratings.csv \
wasb://hd-spark-cluster-2019#hdsparkclusterstorage.blob.core.windows.net/ml-latest/movies.csv
And here is an example of passing the same files using their equivalent https URI
spark-submit \
--class com.nileshgule.movielens.MovieRatingAnalysis \
--master yarn \
--deploy-mode cluster \
--executor-memory 1g \
--name MoviesCsvReader \
--conf "spark.app.id=MovieRatingAnalysis" \
https://hdsparkclusterstorage.blob.core.windows.net/hd-spark-cluster-2019/learning-spark-1.0.jar \
https://hdsparkclusterstorage.blob.core.windows.net/hd-spark-cluster-2019/ml-latest/ratings.csv \
https://hdsparkclusterstorage.blob.core.windows.net/hd-spark-cluster-2019/ml-latest/movies.csv
For hadoop job kindly run the jar from root user . Once you login to HDinsight run the command sudo su - . And the create a folder and place the jar to that folder and run the jar .
I'm currently trying to build a large Docker image and run a shiny application off of it so I can eventually deploy it to a Unix server. The image builds successfully; however, when I go to run the image, the app runs and totally ignores the specified port.
What's even more strange is I first built a small test app, and the instructions from this SO post (Shiny app docker container not loading in browser) worked. I copied the same style I used in the test app into the other Shiny application and now it is not working.
The structure of my Docker image follows a similar structure to what ShinyProxy used on their Github page: https://github.com/openanalytics/shinyproxy-template:
|-- Dockerfile
|-- Rprofile.site
|-- app_stuff
|-- app.R
|-- accessory files called from app.R...
My Dockerfile is below:
# Install R version 3.5.1
FROM r-base:3.5.1
# system libraries of general use - I don't know if these are right ????
RUN apt-get update && apt-get install -y \
default-jdk \
libbz2-dev \
zlib1g-dev \
gfortran \
liblzma-dev \
libpcre3-dev \
libreadline-dev \
xorg-dev \
sudo \
pandoc \
pandoc-citeproc \
libcurl4-gnutls-dev \
libcairo2-dev \
libxt-dev \
libssl-dev \
libssh2-1-dev \
libxml2-dev
RUN R -e "install.packages('remotes');"
RUN R -e "library(remotes); \
remotes::install_version('shiny', version='1.1.0', repos='https://cran.r-project.org/'); \
remotes::install_version('tidyverse', version='1.2.1', repos='https://cran.r-project.org/'); \
remotes::install_version('ggiraph', version='0.6.0', repos='https://cran.r-project.org/'); \
remotes::install_version('plotly', version='4.8.0', repos='https://cran.r-project.org/'); \
remotes::install_version('CausalImpact', version='1.2.3', repos='https://cran.r-project.org/'); \
remotes::install_version('reshape2', version='1.4.3', repos='https://cran.r-project.org/'); \
remotes::install_version('bsts', version='0.8.0', repos='https://cran.r-project.org/'); \
remotes::install_version('xts', version='0.10-2', repos='https://cran.r-project.org/'); \
remotes::install_version('BoomSpikeSlab', version='1.0.0', repos='https://cran.r-project.org/'); \
remotes::install_version('Boom', version='0.8', repos='https://cran.r-project.org/'); \
remotes::install_version('MASS', version='7.3-50', repos='https://cran.r-project.org/'); \
remotes::install_version('dygraphs', version='1.1.1.4', repos='https://cran.r-project.org/'); \
remotes::install_version('prophet', version='0.4', repos='https://cran.r-project.org/'); \
remotes::install_version('rlang', version='0.3.3', repos='https://cran.r-project.org/'); \
remotes::install_version('Rcpp', version='1.0.1', repos='https://cran.r-project.org/'); \
remotes::install_version('zoo', version='1.8-1', repos='https://cran.r-project.org/'); \
remotes::install_version('RJDBC', version='0.2-7.1', repos='https://cran.r-project.org/'); \
remotes::install_version('rJava', version='0.9-10', repos='https://cran.r-project.org/'); \
remotes::install_version('shinyjs', version='1.0', repos='https://cran.r-project.org/'); \
remotes::install_version('DT', version='0.5', repos='https://cran.r-project.org/'); \
remotes::install_version('shinyBS', version='0.61', repos='https://cran.r-project.org/');"
# copy the app to the image
RUN mkdir /root/app_stuff
COPY app_stuff /root/app_stuff
COPY Rprofile.site /usr/lib/R/etc/
EXPOSE 3838
CMD ["R", "-e", "shiny::runApp('/root/app_stuff')"]
My Rprofile.site is:
local({
options(shiny.port = 3838, shiny.host = "0.0.0.0")
})
After building the file using the command
docker build -t price_opt .
and then running the image
docker run -it -p 3838:3838 price_opt
I expected to see the shiny app print out: Listening on http://0.0.0.0:3838, but instead it prints out:
Listening on http://127.0.0.1:6688
Which I'm unable to find on my local machine.
Again, the weirdest thing is this type of setup worked for a smaller shiny app. When I ran that docker run command from above on the smaller app, the app was available under localhost:3838.
Any thoughts on why this is happening? My last thing is it looks like this user on the Shiny Proxy site had a similar issue (https://support.openanalytics.eu/t/shiny-app-listening-on-wrong-host/957). His issue is a typo of sorts, but it does seem to be acting the same way here where the Shiny app is totally ignoring the supplied port number in both Rprofile.site and the docker run command.
EDIT - Solution
Thanks to user #Wil, by changing the last line of the Dockerfile to CMD ["R", "-e", "shiny::runApp('/root/app_stuff', host='0.0.0.0', port=3838)"], the app was able to start up normally on localhost:3838.
Port 3838 is the default port for Shiny Server, but runApp() chooses an available port. It appears R is not picking up your Rprofile.site, so I would just specify the port in your call to runApp():
CMD ["R", "-e", "shiny::runApp('/root/app_stuff',options = list(port = '3838'))"]
What's the easiest way to enable HDFS Caching on EMR ?
More specifically, how to set dfs.datanode.max.locked.memory and increase the "maximum size that may be locked into memory" (ulimit -l) on all nodes ?
The following code seems to work fine for dfs.datanode.max.locked.memory and I could probably write a custom bootstrap to update /usr/lib/hadoop/hadoop-daemon.sh and call ulimit. Is there any better or faster way ?
elastic-mapreduce --create \
--alive \
--plain-output \
--visible-to-all \
--ami-version 3.1.0 \
-a $access_id \
-p $private_key \
--name "test" \
--master-instance-type m3.xlarge \
--instance-group master --instance-type m3.xlarge --instance-count 1 \
--instance-group core --instance-type m3.xlarge --instance-count 10 \
--pig-interactive \
--log-uri s3://foo/bar/logs/ \
--bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop \
--args "--hdfs-key-value,dfs.datanode.max.locked.memory=2000000000" \