Syntax error: SYN0001 despite it working on the Kusto query editor online - azure-data-explorer

Question

I'm building queries in Python and executing them on my Kusto clusters using Kusto client's execute_query method.
I've been hit by the following error: azure.kusto.data.exceptions.KustoApiError: Request is invalid and cannot be processed: Syntax error: SYN0001: I could not parse that, sorry. [line:position=0:0].
However, when debugging, I've taken the query as it is, and ran it on my clusters thru the Kusto platform on Azure.
The query is similar to the following:
StormEvents
| where ingestion_time() > ago(1h)
| summarize
count_matching_regex_State=countif(State matches regex "[A-Z]*"),
count_not_empty_State=countif(isnotempty(State))
| summarize
Matching_State=sum(count_matching_regex_State),
NotEmpty_State=sum(count_not_empty_State)
| project
ratio_State=todouble(Matching_State) / todouble(Matching_State + NotEmpty_State)
| project
ratio_State=iff(isnan(ratio_State), 0.0, round(ratio_State, 3))
Queries are built in Python using string interpolations and such:
## modules.py
def match_regex_query(fields: list, regex_patterns: list, kusto_client):
def match_regex_statements(field, regex_patterns):
return " or ".join(list(map(lambda pattern: f"{field} matches regex \"{pattern}\"", regex_patterns)))
count_regex_statement = list(map(
lambda field: f"count_matching_regex_{field} = countif({match_regex_statements(field, regex_patterns)}), count_not_empty_{field} = countif(isnotempty({field}))", fields))
count_regex_statement = ", ".join(count_regex_statement)
summarize_sum_statement = list(map(lambda field: f"Matching_{field} = sum(count_matching_regex_{field}), NotEmpty_{field} = sum(count_not_empty_{field})", fields))
summarize_sum_statement = ", ".join(summarize_sum_statement)
project_ratio_statement = list(map(lambda field: f"ratio_{field} = todouble(Matching_{field})/todouble(Matching_{field}+NotEmpty_{field})", fields))
project_ratio_statement = ", ".join(project_ratio_statement)
project_round_statement = list(map(lambda field: f"ratio_{field} = iff(isnan(ratio_{field}),0.0,round(ratio_{field}, 3))", fields))
project_round_statement = ", ".join(project_round_statement)
query = f"""
StormEvents
| where ingestion_time() > ago(1h)
| summarize {count_regex_statement}
| summarize {summarize_sum_statement}
| project {project_ratio_statement}
| project {project_round_statement}
"""
clean_query = query.replace("\n", " ").strip()
try:
result = kusto_client.execute_query("Samples", clean_query)
except Exception as err:
logging.exception(
f"Error while computing regex metric : {err}")
result = []
return result
## main.py
#provide your kusto client here
cluster = "https://help.kusto.windows.net"
kcsb = KustoConnectionStringBuilder.with_interactive_login(cluster)
client = KustoClient(kcsb)
fields = ["State"]
regex_patterns = ["[A-Z]*"]
metrics = match_regex_query(fields, regex_patterns, client)
Is there a better way to debug this problem?
TIA!

Answer 1

the query your code generates is invalid, as the regular expressions include characters that aren't properly escaped.
see: the string data type
this is your invalid query (based on the client request ID you provided in the comments):
LiveStream_CL()
| where ingestion_time() > ago(1h)
| summarize count_matching_regex_deviceHostName_s = countif(deviceHostName_s matches regex "^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0,61}[a-zA-Z0-9\$])?$"), count_not_empty_deviceHostName_s = countif(isnotempty(deviceHostName_s)), count_matching_regex_sourceHostName_s = countif(sourceHostName_s matches regex "^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0,61}[a-zA-Z0-9\$])?$"), count_not_empty_sourceHostName_s = countif(isnotempty(sourceHostName_s)), count_matching_regex_destinationHostName_s = countif(destinationHostName_s matches regex "^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0,61}[a-zA-Z0-9\$])?$"), count_not_empty_destinationHostName_s = countif(isnotempty(destinationHostName_s)), count_matching_regex_agentHostName_s = countif(agentHostName_s matches regex "^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0,61}[a-zA-Z0-9\$])?$"), count_not_empty_agentHostName_s = countif(isnotempty(agentHostName_s))
...
whereas this is how it should look like (note the addition of the #s):
LiveStream_CL()
| where ingestion_time() > ago(1h)
| summarize
count_matching_regex_deviceHostName_s = countif(deviceHostName_s matches regex #"^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0,61}[a-zA-Z0-9\$])?$"),
count_not_empty_deviceHostName_s = countif(isnotempty(deviceHostName_s)),
count_matching_regex_sourceHostName_s = countif(sourceHostName_s matches regex #"^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0,61}[a-zA-Z0-9\$])?$"),
count_not_empty_sourceHostName_s = countif(isnotempty(sourceHostName_s)),
count_matching_regex_destinationHostName_s = countif(destinationHostName_s matches regex #"^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0,61}[a-zA-Z0-9\$])?$"),
count_not_empty_destinationHostName_s = countif(isnotempty(destinationHostName_s)),
count_matching_regex_agentHostName_s = countif(agentHostName_s matches regex #"^[a-zA-Z0-9\$]([a-zA-Z0-9\-\_\.\$]{0, 61}[a-zA-Z0-9\$])?$"),
count_not_empty_agentHostName_s = countif(isnotempty(agentHostName_s))
...