I have Telegraf pulling data from OpenWeather API and returning JSON. On occasion there is a missing tag such as rain. If there is no rain predicted then it's not there
My config:
[[inputs.http]] urls =
["https://api.openweathermap.org/data/3.0/onecall?units=metric&lat=51.123456&lon=-0.123456&appid=xxxx&exclude=current,hourly,alerts,minutely"]
interval = "30s"
tagexclude = ["url", "host"]
data_format = "json_v2"
[[inputs.http.json_v2]]
measurement_name = "openweather"
timestamp_path = "daily.0.dt"
timestamp_format = "unix"
[[inputs.http.json_v2.field]]
path = "daily.0.clouds"
[[inputs.http.json_v2.field]]
path = "daily.0.temp.day"
[[inputs.http.json_v2.field]]
path = "daily.0.rain"
I cannot work out how to tell the input that the tag is optional. Either null or 0 would would be fine
Related
I'm trying to make a certain BLF CAN data file.
After I created an arbitary measurements table, I try to encode messages and write on BLF format by folloing codes.
The BLF file was made, however, it doesn't have any data at all.
Please let me know what the problem is.
What I tried :
import cantools
import can
db = cantools.database.load_file('T_Fuel_HB.dbc')
ex_msg = db.get_message_by_name("DEVICE_56604591_0")
time = 0
write_created = can.BLFWriter("sample_created.blf")
for i in range(10) :
r_int = np.random.randint(0, 100)
data_created = ex_msg.encode({'C_1_T_Air' : r_int, 'C_2_T_EG_room' : r_int, 'C_3_T_Pump_room' : r_int, 'C_4_T_Fuel_tank' : r_int})
msg_created = can.Message(timestamp = time, arbitration_id = ex_msg.frame_id, data = data, channel=0)
print(msg_created, r_int)
time += 2
write_created.on_message_received(msg_created)
What I expected :
filename = "VDK14.blf"
log = can.BLFReader(filename)
log = list(log)
for msg in log :
write.on_message_received(msg)
-> When I use the BLF file with existing log data, it's no problem to read the file thru CANanalyzer.
I can receive messages with the inputs.mqtt_consumer telegraf plugin, but it gives me a lot of data in influxdb.
How can I in the telegraf configuration just count the number of received bytes and messages and report that to influx db?
# Configuration for telegraf agent
[agent]
interval = "20s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
[[outputs.influxdb_v2]]
urls = ["XXXXXXXXXXXXXXXX"]
token = "$INFLUX_TOKEN"
organization = "XXXXXXXXXXXXXXX"
bucket = "XXXXXXXXXXXXXXX"
[[inputs.mqtt_consumer]]
servers = ["tcp://XXXXXXXXXXXXXXXXXXXXX:1883"]
topics = [
"#",
]
data_format = "value"
data_type = "string"
I tried to google around but din't find any clear ways to do it.
I just want number of bytes and messages received each minute for the selected topic
I did not manage to receive all the messages and count them, but I found a solution where I can get the data from the broker. Not exactly what I asked for but fine for what I need.
topics = [
"$SYS/broker/load/messages/received/1min",
"$SYS/broker/load/messages/sent/1min",
]
...
data_format = "value"
data_type = "float"
I am trying to programmatically update my Figshare repository using rapiclient.
Following the answer to this question, I managed to authenticate and see my repository by:
library(rapiclient)
library(httr)
# figshare repo id
id = 3761562
fs_api <- get_api("https://docs.figshare.com/swagger.json")
header <- c(Authorization = sprintf("token %s", Sys.getenv("RFIGSHARE_PAT")))
fs_api <- list(operations = get_operations(fs_api, header),
schemas = get_schemas(fs_api))
reply <- fs_api$operations$article_files(id)
I also managed to delete a file using:
fs_api$operations$private_article_file_delete(article_id = id, file_id = F)
Now, I would like to upload a new file to the repository. There seem to be two methods I need:
fs_api$operations$private_article_upload_initiate
fs_api$operations$private_article_upload_complete
But I do not understand the documentation. According to fs_api$operations$private_article_upload_initiate help:
> fs_api$operations$private_article_upload_initiate
private_article_upload_initiate
Initiate Upload
Description:
Initiate new file upload within the article. Either use link to
provide only an existing file that will not be uploaded on figshare
or use the other 3 parameters(md5, name, size)
Parameters:
link (string)
Url for an existing file that will not be uploaded on figshare
md5 (string)
MD5 sum pre computed on the client side
name (string)
File name including the extension; can be omitted only for linked
files.
size (integer)
File size in bytes; can be omitted only for linked files.
What does "file that will not be uploaded on Figshare" mean? How would I use the API to upload a local file ~/foo.txt?
fs_api$operations$private_article_upload_initiate(link='~/foo.txt')
returns HTTP 400.
I feel like I sent you down a bad path with my previous answer because I am not sure how to edit some of the api endpoints when using rapiclient. For example, the corresponding endpoint for fs_api$operations$private_article_upload_initiate() will be https://api.figshare.com/v2/account/articles/{article_id}/files, and I am not sure how to substitute for {article_id} prior to sending the request.
You may have to define your own client for operations you cannot get working any other way.
Here is an example of uploading a file to an existing private article as per the goal of your question.
library(httr)
# id of previously created figshare article
my_article_id <- 99999999
# make example file to upload
my_file <- tempfile("my_file", fileext = ".txt")
writeLines("Hello World!", my_file)
# Step 1 initiate upload
# https://docs.figshare.com/#private_article_upload_initiate
r <- POST(
url = sprintf("https://api.figshare.com/v2/account/articles/%s/files", my_article_id),
add_headers(c(Authorization = sprintf("token %s", Sys.getenv("RFIGSHARE_PAT")))),
body = list(
md5 = tools::md5sum(my_file)[[1]],
name = basename(my_file),
size = file.size(my_file)
),
encode = "json"
)
initiate_upload_response <- content(r)
# Step 2 single file info (get upload url)
# https://docs.figshare.com/#private_article_file
r <- GET(url = initiate_upload_response$location,
add_headers(c(Authorization = sprintf("token %s", Sys.getenv("RFIGSHARE_PAT"))))
)
single_file_response <- content(r)
# Step 3 uploader service (get number of upload parts required)
# https://docs.figshare.com/#endpoints
r <- GET(url = single_file_response$upload_url,
add_headers(c(Authorization = sprintf("token %s", Sys.getenv("RFIGSHARE_PAT"))))
)
upload_service_response <- content(r)
# Step 4 upload parts (this example only has one part)
# https://docs.figshare.com/#endpoints_1
r <- PUT(url = single_file_response$upload_url, path = 1,
add_headers(c(Authorization = sprintf("token %s", Sys.getenv("RFIGSHARE_PAT")))),
body = upload_file(my_file)
)
upload_parts_response <- content(r)
# Step 5 complete upload (after all part uploads are successful)
# https://docs.figshare.com/#private_article_upload_complete
r <- POST(
url = initiate_upload_response$location,
add_headers(c(Authorization = sprintf("token %s", Sys.getenv("RFIGSHARE_PAT"))))
)
complete_upload_response <- content(r)
I have the following problem: the following code works perfectly:
inventJournalTrans.clear();
inventJournalTrans.initFromInventJournalTable(inventJournalTable);
inventJournalTrans.ItemId = "100836M";
frominventDim.InventLocationId="SD";
frominventDim.wMSLocationId = '11_RECEPTION';
fromInventDim.InventSizeId = '1000';
fromInventDim.inventBatchId = 'ID057828-CN';
ToinventDim.InventLocationId = "SD";
ToInventDim.wMSLocationId = '11_A2';
ToInventDim.InventSizeId = '1000';
ToInventDim.inventBatchId = 'T20/0001/1';
ToinventDim = InventDim::findOrCreate(ToinventDim);
frominventDim = InventDim::findOrCreate(frominventDim);
inventJournalTrans.InventDimId = frominventDim.inventDimId;
inventJournalTrans.initFromInventTable(InventTable::find("100836M"));
inventJournalTrans.Qty = -0.5;
inventJournalTrans.ToInventDimId = ToinventDim.inventDimId;
inventJournalTrans.CostAmount = InventJournalTrans.calcCostAmount(-abs(any2real(strReplace('-0.5',',','.'))));
inventJournalTrans.TransDate = SystemDateget();
inventJournalTrans.insert();
inventJournalCheckPost = InventJournalCheckPost::newJournalCheckPost(JournalCheckpostType::Post,inventJournalTable);
inventJournalCheckPost.parmThrowCheckFailed(_throwserror);
inventJournalCheckPost.parmShowInfoResult(_showinforesult);
inventJournalCheckPost.run();
It creates transfer journal line correctly and posts the transfer journal successfully.
My requirement is to import journal lines from a csv file. I wrote the following code:
inventJournalTrans.clear();
inventJournalTrans.initFromInventJournalTable(inventJournalTable::find(_journalID));
InventJournaltrans.ItemId = conpeek(_filerecord,4);
inventDim_From.InventLocationId = 'SD';
inventDim_From.wMSLocationId = '11_RECEPTION';
InventDim_from.InventSizeId = conpeek(_fileRecord,11);
InventDim_From.inventBatchId = strfmt("%1",conpeek(_fileRecord,5));
InventDim_To.InventLocationId = 'SD';
inventDim_To.wMSLocationId = strfmt("%1",conpeek(_fileRecord,10));
InventDim_To.InventSizeId = conpeek(_fileRecord,11);
InventDim_To.inventBatchId = strfmt("%1",conpeek(_fileRecord,6));
InventDim_From = InventDim::findOrCreate(inventDim_From);
inventDim_To = InventDim::findOrCreate(inventDim_To);
InventJournalTrans.InventDimId = inventDim_From.inventDimId;
InventJournalTrans.initFromInventTable(InventTable::find(conpeek(_filerecord,4)));
inventJournalTrans.Qty = -abs(any2real(strReplace(conpeek(_fileRecord,8),',','.')));
inventJournalTrans.ToInventDimId = inventDim_To.inventDimId;
InventJournalTrans.CostAmount = InventJournalTrans.calcCostAmount(-abs(any2real(strReplace(conpeek(_fileRecord,8),',','.'))));
inventJournalTrans.TransDate = str2date(conpeek(filerecord,9),123);
InventJournalTrans.insert();
I have the following error when I use the insert() method : size does not exists for the itemId. When I have a look in inventSize table for my itemId, the size exists, I thought it was an inventDimId problem in inventJournalTrans but they're strictly similar to the first code exemple. All my datas are the same as the first exemple but are not hard-coded and come from reading my csv file.
I spend a lot of time debugging and found nothing wrong but error message remains
I'm using Dynamics AX V4 SP1.
Thanks a lot for any help.
When you read from files always trim trailing spaces. You can do that using the strRtrim function.
Like so:
InventDim_To.InventSizeId = strRtrim(conpeek(_fileRecord,11));
I am currently doing a project on football players. I am trying to scrape some public comments on football players for sentiment analysis. However I can't seem to scrape the comment. Any help would be MUCH appreciated. It is the comments part I can't seem to do. Weirdly enough I had it working but then it stopped and I cant seem to get scraping comments again. The website I am scraping from is : https://sofifa.com/player/192985/kevin-de-bruyne/200025/
likes = []
dislikes = []
follows = []
comments = []
driver_path = '/Users/niallmcnulty/Desktop/GeneralAssembly/Lessons/DSI11-lessons/week05/day2_web_scraping_and_apis/web_scraping/selenium-examples/chromedriver'
driver = webdriver.Chrome(executable_path=driver_path)
# i = 0
for url in tqdm_notebook(urls):
driver.get(url)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
sleep(0.2)
soup1 = BeautifulSoup(driver.page_source,'lxml')
try:
dislike = soup1.find('button', attrs = {'class':'bp3-button bp3-minimal bp3-intent-danger dislike-btn need-sign-in'}).find('span',{'class':'count'}).text.strip()
dislikes.append(dislike)
except:
pass
try:
like = soup1.find('button', attrs = {'class':'bp3-button bp3-minimal bp3-intent-success like-btn need-sign-in'}).find('span',{'class':'count'}).text.strip()
likes.append(like)
except:
pass
try:
follow = soup1.find('button', attrs = {'class':'bp3-button bp3-minimal follow-btn need-sign-in'}).find('span',{'class':'count'}).text.strip()
follows.append(follow)
except:
pass
try:
comment = soup1.find_all('p').text[0:10]
comments.append(comment)
except:
pass
# i += 1
# if i % 5 == 0:
# sentiment = pd.DataFrame({"dislikes":dislikes,"likes":likes,"follows":follows,"comments":comments})
# sentiment.to_csv('/Users/niallmcnulty/Desktop/GeneralAssembly/Lessons/DSI11-lessons/projects/cap-csv/sentiment.csv')
sentiment_final = pd.DataFrame({"dislikes":dislikes,"likes":likes,"follows":follows,"comments":comments})
# df_sent = pd.merge(df, sentiment, left_index=True, right_index=True)
The comments section is dynamically loaded. you can try to capture it using the driver,
try:
comment_elements = driver.find_elements_by_tag_name('p')
for comment in comment_elements:
comments.append(comment.text)
except:
pass
print(Comments)