unable to store json in single line inside azure blob storage using azure function - json.net

I have a requirement to store json in a single line(without any formatting) inside a blob storage file. I am using azure function with Newtonsoft.JSon properties for some manipulation purpose and then writing to a blob . But when I try to using JToken.Parse I am getting exception or internal server error.Below is the code I am using:
#r "Newtonsoft.Json"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using System.Linq;
using System.Threading.Tasks;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public static async Task<IActionResult> Run(HttpRequest req,TextWriter outputBlob,ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
log.LogInformation($"Response is {requestBody}");
dynamic jObject = JsonConvert.DeserializeObject(requestBody);
JToken jCategory = jObject;
var clus = jCategory["clusters"];
foreach(JObject item in clus)
{
var custom_tag=item["custom_tags"];
var app_logical_name = item.SelectToken("custom_tags.app_name");
item.SelectToken("init_scripts_safe_mode").Parent.AddAfterSelf(new JProperty("app_logical_name",app_logical_name));
}
var clus2 = JsonConvert.SerializeObject(jCategory,Formatting.None);
//var clus_new=JArray.Parse(clus).toString(Newtonsoft.Json.Formatting.None);
outputBlob.Write(clus2);
// outputBlob.Write(clus_new);
return new OkObjectResult(requestBody);
}
I have tried both ways but both are giving runtime errors. I just need to put the json in a single line(without any formatting) and write to blob.Can you please help me in this ?

This is the structure on my side:
{
"clusters":[
{
"custom_tags":{
"app_name": "appname1"
},
"init_scripts_safe_mode":{
"xxx": "yyy"
}
},
{
"custom_tags":{
"app_name": "appname2"
},
"init_scripts_safe_mode":{
"xxx2": "yyy2"
}
}
],
"test":"333"
}
And this is my code:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
namespace FunctionApp3
{
public static class Function1
{
[FunctionName("Function1")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
log.LogInformation($"Response is {requestBody}");
dynamic jObject = JsonConvert.DeserializeObject(requestBody);
JToken jCategory = jObject;
var clus = jCategory["clusters"];s2 = JsonConvert.SerializeObject(jCategory, Formatting.None);
foreach (JObject item in clus)
{
var custom_tag = item["custom_tags"];
var app_logical_name = item.SelectToken("custom_tags.app_name");
var xxx = item.SelectToken("init_scripts_safe_mode");
xxx.Parent.AddAfterSelf(new JProperty("app_logical_name", app_logical_name));
log.LogInformation(JsonConvert.SerializeObject(custom_tag, Formatting.None));
log.LogInformation(JsonConvert.SerializeObject(app_logical_name, Formatting.None));
}
return new OkObjectResult(clus);
}
}
}
It seems no problem:
If you get the server side error, please check the details log to get where is the error comes from.
The 500 error is not helpful to solve this problem, you need to check the specific error of the azure function. You can use application insights to get the details error. The function must configure the corresponding application insights before you can view the log on the portal.
So you need to configure an application insights to your function app like this:
Then your function app will restart.
Of course, you can also go to kudu to view:
First, go to advanced tools, then click 'GO',
Then After you go to kudu, click Debug Console -> CMD -> LogFiles -> Application -> Functions -> yourtriggername. You will find log file there.
If you are based on linux OS, after go to kudu, just click 'log stream'(this is not supportted to consumption plan for linux.).

Related

How to override the connectionstring value of AzureWebJobsStorage to a new value in .net core function app

I have to specify the storage account connections attribute based on some condition
eg:
if(condition=true)
[StorageAccount("ConnectionString1")]
else
[StorageAccount("ConnectionString2")]
Do we have some conditional attributes to achieve this in .net?
One of the workarounds I did to override the Azure Function App Storage Accounts is:
Created a .NET Core 6 Azure Function App and one extra storage account in which both storage account connection strings are noted in My Function App local.settings.json file like below:
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "Get_Conn_String_from_AzureStorageAcc_AccessKeys",
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
"AzureStorageAccount02": "<Get_Conn_String_from_AzureStorageAcc_AccessKeys>"
}
}
To override the different storage account based on condition from the Azure Function, I have written the following code:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
namespace KrishNet6CoreFunctionApp
{
public static class Function1
{
[FunctionName("Function1")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
if (name == "HariKrishna")
{
var constring1 = Environment.GetEnvironmentVariable("AzureWebJobsStorage");
log.LogInformation("Function Execution logs will be stored in 1st storage account.");
}
else
{
var constring1 = Environment.GetEnvironmentVariable("AzureStorageAccount02");
log.LogInformation("Function Execution logs will be stored in 2nd storage account.");
}
string responseMessage = string.IsNullOrEmpty(name)
? "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."
: $"Hello, {name}. This HTTP triggered function executed successfully.";
return new OkObjectResult(responseMessage);
}
}
}
Result:
If the parameter name value is different, then utilize the 2nd Storage account to store logs, perform any other operations/activities:

How to fix error: "Google.Apis.Auth: At least one client secrets (Installed or Web) should be set " for YouTubeAPI

I have created an HTTP trigger azure function, which holds the code (below) to upload a video to YouTube automatically. Source: (https://developers.google.com/youtube/v3/docs/videos/insert).
When I try to run the app locally using visual studio, I am getting the following error:
Executed 'Function1' (Failed, Id=d601d64a-2f2c-4f8a-8053-a2f33ca21dbc)
System.Private.CoreLib: Exception while executing function: Function1.
Google.Apis.Auth: At least one client secrets (Installed or Web)
should be set
It looks like a Google Authentication error, but I am unsure as to how to get this fixed and I see that YouTube API does not support Service account? How can this issue be fixed, is there a get around to this? Thanks in advance.
C# Code:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Upload;
using Google.Apis.YouTube.v3.Data;
using System.Reflection;
using Google.Apis.YouTube.v3;
using Google.Apis.Services;
using System.Threading;
namespace UploadVideo
{
public static class Function1
{
[FunctionName("Function1")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
log.LogInformation("YouTube Data API: Upload Video");
log.LogInformation("==============================");
try
{
await Run();
}
catch (AggregateException ex)
{
foreach (var e in ex.InnerExceptions)
{
log.LogInformation("Error: " + e.Message);
}
}
return new OkObjectResult($"Video Processed..");
}
private static async Task Run()
{
UserCredential credential;
using (var stream = new FileStream("client_secrets.json", FileMode.Open, FileAccess.Read))
{
credential = await GoogleWebAuthorizationBroker.AuthorizeAsync(
GoogleClientSecrets.Load(stream).Secrets,
// This OAuth 2.0 access scope allows an application to upload files to the
// authenticated user's YouTube channel, but doesn't allow other types of access.
new[] { YouTubeService.Scope.YoutubeUpload },
"user",
CancellationToken.None
);
}
var youtubeService = new YouTubeService(new BaseClientService.Initializer()
{
HttpClientInitializer = credential,
ApplicationName = Assembly.GetExecutingAssembly().GetName().Name
});
var video = new Video();
video.Snippet = new VideoSnippet();
video.Snippet.Title = "Default Video Title";
video.Snippet.Description = "Default Video Description";
video.Snippet.Tags = new string[] { "tag1", "tag2" };
video.Snippet.CategoryId = "22"; // See https://developers.google.com/youtube/v3/docs/videoCategories/list
video.Status = new VideoStatus();
video.Status.PrivacyStatus = "unlisted"; // or "private" or "public"
var filePath = #"C:\Users\Peter\Desktop\audio\test.mp4"; // Replace with path to actual movie file.
using (var fileStream = new FileStream(filePath, FileMode.Open))
{
var videosInsertRequest = youtubeService.Videos.Insert(video, "snippet,status", fileStream, "video/*");
videosInsertRequest.ProgressChanged += videosInsertRequest_ProgressChanged;
videosInsertRequest.ResponseReceived += videosInsertRequest_ResponseReceived;
await videosInsertRequest.UploadAsync();
}
}
private static void videosInsertRequest_ProgressChanged(Google.Apis.Upload.IUploadProgress progress)
{
switch (progress.Status)
{
case UploadStatus.Uploading:
Console.WriteLine("{0} bytes sent.", progress.BytesSent);
break;
case UploadStatus.Failed:
Console.WriteLine("An error prevented the upload from completing.\n{0}", progress.Exception);
break;
}
}
private static void videosInsertRequest_ResponseReceived(Video video)
{
Console.WriteLine("Video id '{0}' was successfully uploaded.", video.Id);
}
}
}
It looks like you were trying to use the service account to do the OAuth2 web server flow, which wont work. The correct code form creating a service account credential is as follows.
GoogleCredential credential;
using (var stream = new FileStream(serviceAccountCredentialFilePath, FileMode.Open, FileAccess.Read))
{
credential = GoogleCredential.FromStream(stream)
.CreateScoped(scopes);
}
Note
as i have mentioned in your other questions the YouTube API does NOT support service account authentication. You must use Oauth2 and i am not convinced this can be done inside of azure functions. As there is no way to spawn the web browser window to request authorization of the user.

How to make code execute automatically every minute in asp.net mvc5 my websites

I have this code I have done using the application console and I want to use it inside my website using asp.net mvc5 in order to store the data it brings from the RSS feeds and store it in a database
using System;
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using System.Xml.Linq;
using System.Text;
using System.Threading.Tasks;
using System.ComponentModel.DataAnnotations;
using System.Data.Entity;
using CodeHollow.FeedReader;
using CodeHollow.FeedReader.Feeds;
using System.Data;
using System.Data.SqlClient;
using System.Data.Entity.Core;
using System.Data.Entity.Infrastructure;
namespace RssFeedBackEnd
{
public class Program
{
public static void Main(string[] args)
{
RssFeedDB db = new RssFeedDB();
var qry = (from c in db.Links select c);
foreach (var li in qry)
{
try
{
var feed = FeedReader.Read(li.adressLink);
var item = feed.Items;
if (feed.Type.ToString() == "Rss_2_0")
{
//New n = new New();
foreach (var items in item)
{
string v = items.Id;
int h = validate(v);
if (h == 0)
{
try
{
New n = new New();
n.TitleNews = items.Title;
n.LinkNews = items.Link;
n.TitlePage = feed.Title;
n.LinkPage = feed.Link;
n.Linkimg = items.SpecificItem.Element.Element("enclosure").Attribute("url").Value;
n.IDurl = items.Id;
n.Pubdate = DateTime.Now;
n.EntryTime = DateTime.Now;
n.Description = items.Description;
n.IdCatogrey = li.CategorayID;
//n.NameCatogrey = li.Category.NameCategory.ToString();
db.News.Add(n);
}
catch { }
}
}
}
}
catch { }
}
db.SaveChanges();
}
static int validate(string n) {
int a = 0;
RssFeedDB db = new RssFeedDB();
var qry = (from m in db.News select m);
foreach (var nws in qry)
{
if (nws.IDurl == n) { a =1; break; }
}
return a;
}
}}
I hope to help me how to put it inside the web application and do his work every minute .
The idea of ​​my project is to fetch data (news) from RSS feeds links and store them in the database to display on my website.
I hope I've shown you very well what I want to do.
A web application is fundamentally unsuited for this type of task. A website's job is to respond to requests from others (usually people, via browsers) as and when they occur. It does not run automatically at specific times.
To achieve your goal, on your server you need either a Windows Service with a timer in it (which then executes the necessary code every time the timer expires), or a Windows Scheduled Task which triggers a specific application at regular intervals.
Since you already have a Console application which does the job you need, then the simplest solution for you is probably to set up a Windows Scheduled Task which executes the console application regularly.
N.B. You can of course still have a separate website which can display the data that the console application has saved in the database.
P.S. There are extensions to ASP.NET such as Hangifire and Quartz.NET which can add scheduling capabilities to a web application, but they are probably overkill for what you want to do in this case.

How to send an XML to an asp.net Web api call?

I am trying to make a web API Post method call as follows but it not working as expected,xmlcontent seems OK but somehow the formatting seems messed up when the request is being sent and the response throws an error ,I double checked the XML from python and it works,is there a better way to create and send the XML?what am I doing wrong?
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Net.Http;
namespace WebApiXML
{
public class Program
{
static void Main(string[] args)
{
testWCF2(); //Or whatever
Console.ReadLine();
}
public static async Task testWCF2()
{
string xmlcontent = #"<SoftwareProductBuild>
<BuildSource>DATASOURCE</BuildSource>
<BuiltBy>username1</BuiltBy>
<CreatedBy>username1</CreatedBy>
<Name>username1_1959965_1969310_524f7fef-5b37-11e7-b4ee-f0921c133f10_UL.AB.1.2_test2</Name>
<Status>Approved</Status>
<BuiltOn>2017-06-27T06:20:30.275690</BuiltOn>
<Tag>username1_1959965_1969310_524f7fef-5b37-11e7-b4ee-f0921c133f10_test2</Tag>
<Keywords>
<KeywordInfo>
<Name>subystem</Name>
</KeywordInfo>
</Keywords>
<SoftwareImageBuilds>
<SoftwareImageBuild>
<Type>LA</Type>
<Name>username1_1959965_1969310_524f7fef-5b37-11e7-b4ee-f0921c133f10_UL.AB.1.2_test2</Name>
<Location>\\location1\data1\PRECOMMIT_OS_DEF</Location>
<Variant>PRECOMMIT_OS_DEF</Variant>
<LoadType>Direct</LoadType>
<Target>msm8998</Target>
<SoftwareImages>
<SoftwareImage>
<Name>UL.AB.1.2</Name>
</SoftwareImage>
</SoftwareImages>
</SoftwareImageBuild>
</SoftwareImageBuilds>
</SoftwareProductBuild>";
#region using
using (var client = new System.Net.Http.HttpClient())
{
var response = await client.PostAsXmlAsync("http://server:8100/api/SoftwareProductBuild", xmlcontent);
if (!response.IsSuccessStatusCode)
{
//throw new InvalidUriException("Some error with details.");
Console.WriteLine(response);
}
Console.WriteLine("Printing DEV Pool Response\n");
}
#endregion
//return null;
}
}
}
PostAsXmlAsync will try to serialize the object passed to it. So you have a string that contains XML and then try to post the string as XML(Double serialization).
Use StringContent, giving it the XML string value and set the content type to appropriate media type, then post it. i.e. client.PostAsync(url, content)
using (var client = new System.Net.Http.HttpClient()) {
var url = "http://server:8100/api/SoftwareProductBuild";
var content = new StringContent(xmlcontent, Encoding.UTF8, "application/xml");
var response = await client.PostAsync(url, content);
if (response.IsSuccessStatusCode) {
var responseBody = await response.Content.ReadAsStringAsync();
Console.WriteLine("Printing DEV Pool Response\n");
Console.WriteLine(responseBody);
} else {
Console.WriteLine(string.Format("Bad Response {0} \n", response.StatusCode.ToString()));
}
}

Multiple threads in a webservice

I have a working web service which On load contacts different websites and scrapes relevant information from them. As the requirements grew so did the number of httpwebrequests.
Right now I'm not using any asynchronous requests in the web service - Which means that ASP.net renders one request at a time. This obviously became a burden as one request to the webservice itself can take up to 2 minutes to complete.
Is there a way to convert all these httpwebreqeusts inside the webservice to multi-threaded?
What would be the best way to achieve this?
Thanks!
If you are working with .Net V4+, you can use the Parallel library or task library which allow easily to do such things.
If you call all your web services using the same way (assuming all web services respects the same WSDL, just differing urls, you can use something like this) :
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text.RegularExpressions;
namespace ConsoleApplication2
{
class Program
{
private const string StartUrl = #"http://blog.hand-net.com";
private static void Main()
{
var content = DownloadAsString(StartUrl);
// The "AsParallel" here is the key
var result = ExtractUrls(content).AsParallel().Select(
link =>
{
Console.WriteLine("... Fetching {0} started", link);
var req = WebRequest.CreateDefault(new Uri(link));
var resp = req.GetResponse();
var info = new { Link = link, Size = resp.ContentLength};
resp.Close();
return info;
}
);
foreach (var linkInfo in result)
{
Console.WriteLine("Link : {0}", linkInfo.Link);
Console.WriteLine("Size : {0}", linkInfo.Size);
}
}
private static string DownloadAsString(string url)
{
using (var wc = new WebClient())
{
return wc.DownloadString(url);
}
}
private static IEnumerable<string> ExtractUrls(string content)
{
var regEx = new Regex(#"<a\s+href=""(?<url>.*?)""");
var matches = regEx.Matches(content);
return matches.Cast<Match>().Select(m => m.Groups["url"].Value);
}
}
}
This small program first download an html page, then extract all href. This produces an array of remote files.
the AsParralel here allow to run the content of the select in a parallel way.
This code does not have error handling, cancellation feature but illustrate the AsParallel method.
If you can't call all your webservices in the same way, you can also use something like this :
Task.WaitAll(
Task.Factory.StartNew(()=>GetDataFromWebServiceA()),
Task.Factory.StartNew(()=>GetDataFromWebServiceB()),
Task.Factory.StartNew(()=>GetDataFromWebServiceC()),
Task.Factory.StartNew(()=>GetDataFromWebServiceD()),
);
This code will add 4 tasks, that will be run "when possible". The WaitAll method will simply wait for all task to be completed before returning.
By when possible I mean when a slot in the thread pool is free. When using the Task library, there is by default one thread pool per processor core. If you have 100 tasks, the 100 taks will be processed by 4 worker threads on a 4 core computer.

Resources