I have some steps in my ADF pipeline that could use some comments. As an example, I have an aggregate step that creates a new column with comma separated values:
/* initialize #acc as empty string, add #item if #acc is empty, otherwise add: , #item */
reduce(collect(value), '', #acc + iif(#acc == '', '', ', ') + #item, #result)
In the ARM this looks like (ARM template was generated from ADFs export template function):
...
"script": "[concat('
source(
...
commaseparated_values = /* initialize #acc as empty string, add #item if #acc is empty, otherwise add: , #item */\r\nreduce(collect(value), '', #acc + iif(#acc == '', '', ', ') + #item, #result),
...
')]"
When I deploy ADF using this template however, the comment is gone from the expression:
I tried escaping the / with a \ in the ARM, but the comment still does not show up in ADF expression builder. How can I get the comments in expressions deployed via ARM?
I tried escaping the / in the ARM, but the comment still does not show
up in ADF expression builder. How can I get the comments in
expressions deployed via ARM?
One of the workaround could be resolve the above issue
Instead of using /*.../* we can use this /* ....... */ for commenting expresion .
For more information please refer this Microsoft Documentation: Build expressions in mapping data flow
Use version 2.3.0 or later if you're deploying the ARM Templates using Azure CLI and specify the --handle-extended-json-format switch.
In VS Code, if you see the ARM Template word at bottom which means it detected the language mode, use the Inline comments which are valid.
Also try setting up the comments in multi-line syntax once to see multi-line comments are reflecting or not.
For more information related to multi-line comment sytanx, refer this Microsoft Documentation.
In Research, many issues related to comments in ARM Templates were due to improper format of Json, previous Azure CLI versions are not supported comments because JSON.Net handles comments so solved using --handle-extended-json-format as I have mentioned in the starting point.
References:
https://github.com/Azure/azure-resource-manager-schemas/issues/851
https://github.com/Azure/azure-xplat-cli/issues/2112
Related
I'm trying to write a custom injection for Neovim's Treesitter that will highlight tagged template literals as SQL code.
The code in question looks like this:
import sql from "postgres"
const query = sql` SELECT * FROM my_table`
I'd like to tell Treesitter to parse this tagged template literal as SQL code. This is possible using Treesitter's custom injections; in fact the library uses this same technique to inject syntax for GraphQL tagged template literals, which I believe happens in this file here.
I've written a configuration file at ~/.config/nvim/after/queries/ecma/injections.scm that's quite similar to the GraphQL query, and although Treesitter is recognizing the file, the injection isn't working at all. Does anyone have a solution to this? Ideally, Treesitter would see the sql tagged template literal and interpret the body of the function with the SQL parser.
to make tree sitter use injections in your after/ folder, your injections.scm file should contain ; extends as first line.
Took me many hours to figure it out, it's now documented right here
Similar to what #brotify-force said, but my own SQL injection for Python was not working until I realized what the issue was.
~/.config/nvim/after/queries/python/injections.scm
; extends
(assignment
left: (identifier) #_id (#match? #_id "sql")
right: (string) #sql
)
My original query used #id, but it seems like if you have any captures not starting with an underscore besides the language capture, the Neovim Treesitter machinery gets confused.
I've been trying to do something similar but for template strings that just happen to contain SQL. I was inspired by a VSCode plugin that syntax highlights any template string that has a comment like /*sql*/ in front of it. Here's what I came up with.
; extends
(
(comment) #_comment-text
(#match? #_comment-text "sql")
(template_string) #ql
)
I do wish there was a proper sql module for tree-sitter, and it looks like one is in the works, but the ql module will do for now.
I have recently been trying to upgrade flyway from v4 to v6.5.3. During the process, I faced an issue related to placeholders.
The migration fails with the following error.
ERROR: Unable to parse statement in
D:\Softwares\flyway-commandline-6.5.5-windows-x64\flyway-6.5.5\sql\V3__mysql-7.0.sql
at line 101 col 1. See
https://flywaydb.org/documentation/knownparserlimitations for more
information: No value provided for placeholder: ${'), NOW(), NOW())}.
Check your configuration! Caused by: No value provided for
placeholder: ${'), NOW(), NOW())}. Check your configuration!
SQL,
insert into `configuration`(key, value, created_date , updated_date) values('LOG_LOCATION', REPLACE('${MY_LOG_LOCATION}','#{','${'), NOW(), NOW());
To resolve the above failure, I replaced "${" with "$\{", but this is something I didn't look out for.
On debugging the flyway code, I saw it fails in parsing (during validation, method org.flywaydb.core.internal.parser.Parser.readToken()) of the SQL.
Why is it considering the 3rd argument of REPLACE function as a placeholder?
Note: The above SQL works in Flyway v4
If you read the documentation here, you can see that flyway uses a particular syntax for it's placeholders, ${somestring}. In your code, you have a ${, but it's not defining a placeholder. While Flyway is a pretty sophisticated tool, the mechanism here is simply using string matching. If you want to, you can modify your Flyway instance to use different escape characters for the placeholders. This could be handy if you have to have strings in your code that match that ${ syntax. You can read about that here. Scroll down to where you define the PlaceHolderPrefix and PlaceHolderSuffix. Change those to a different combination of values that are not used in your code and you should be fine.
Despite setting flyway.placeholderReplacement=false I keep seeing error about no value provided for placeholder expression in sql by Flyway
ERROR: Unexpected error
org.flywaydb.core.api.FlywayException: No value provided for placeholder expressions: & conditions. Check your configuration!
at org.flywaydb.core.internal.database.oracle.pro.SQLPlusPlaceholderReplacer.replacePlaceholders(SQLPlusPlaceholderReplacer.java:132)
at org.flywaydb.core.internal.util.line.PlaceholderReplacingLine.getLine(PlaceholderReplacingLine.java:36)
at org.flywaydb.core.internal.database.ExecutableSqlScript.extractStatements(ExecutableSqlScript.java:156)
at org.flywaydb.core.internal.database.ExecutableSqlScript.(ExecutableSqlScript.java:133)
at org.flywaydb.core.internal.database.oracle.OracleSqlScript.(OracleSqlScript.java:61)
at org.flywaydb.core.internal.database.oracle.OracleDatabase.doCreateSqlScript(OracleDatabase.java:126)
at org.flywaydb.core.internal.database.Database.createSqlScript(Database.java:163)
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.getSqlScript(SqlMigrationExecutor.java:96)
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.executeInTransaction(SqlMigrationExecutor.java:109)
at org.flywaydb.core.internal.command.DbMigrate.isExecuteGroupInTransaction(DbMigrate.java:312)
at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:275)
at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:244)
at org.flywaydb.core.internal.command.DbMigrate.access$100(DbMigrate.java:53)
at org.flywaydb.core.internal.command.DbMigrate$2.call(DbMigrate.java:163)
at org.flywaydb.core.internal.command.DbMigrate$2.call(DbMigrate.java:160)
at org.flywaydb.core.internal.database.Connection$1.call(Connection.java:145)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:74)
at org.flywaydb.core.internal.database.Connection.lock(Connection.java:141)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:150)
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:160)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:138)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:947)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:910)
at org.flywaydb.core.Flyway.execute(Flyway.java:1238)
at org.flywaydb.core.Flyway.migrate(Flyway.java:910)
at org.flywaydb.commandline.Main.executeOperation(Main.java:161)
at org.flywaydb.commandline.Main.main(Main.java:108)
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Using spring boot, in application.yml.
Add the below placeholderReplacement: false
flyway:
baseline-on-migrate: false
sql-migration-prefix: V
table: migration
placeholderReplacement: false
This was happening, because i had in my .SQL migration file, HTML code with ${name}.
So it was trying to replace that! and i want it to be as is, to be inserted in the database.
Summary: in my case i want it always disabled, as i have no use of it
References: Possible Usages In Different Configuration
flyway.placeholderReplacement=false is only for Flyway placeholders, not SQL*Plus placeholders.
To disable SQL*Plus-specific placeholders, you must include SET DEFINE OFF in your script.
I think, best way to resolve this error - it's override default prefix and suffix
flyway.placeholderPrefix=$${
flyway.placeholderSuffix=}
because disabling this functionality may be unacceptably for some reasons: using flyway variables, for instance.
If your use case is similar to mine (you cannot disable placeholders and you cannot change the prefix), an alternative way is to break the placeholder value. Let's says you SQL is:
UPDATE table SET value = '${variable}';
You can avoid FlyWay picking up the placeholder by using something like:
UPDATE table SET value = '$' + '{variable}';
you can encode the whole string with UTF-8, then decode it when you need to use it
I want to give my users the possibility to create document templates (contracts, emails, etc.)
The best option I figured out would be to store these document templates in mongo (maybe I'm wrong...)
I've been searching for a couple of hours now but I can't figure out how to render these document template with their data context.
Example:
Template stored in Mongo: "Dear {{firstname}}"
data context: {firstname: "Tom"}
On Tom's website, He should read: "Dear Tom"
How can I do this?
EDIT
After some researches, I discovered a package called spacebars-compiler that brings the option to compile to the client:
meteor add spacebars-compiler
I then tried something like this:
Template.doctypesList.rendered = ->
content = "<div>" + this.data.content + "</div>"
template = Spacebars.compile content
rendered = UI.dynamic(template,{name:"nicolas"})
UI.insert(rendered, $(this).closest(".widget-body"))
but it doesn't work.
the template gets compiled but then, I don't know how to interpret it with its data context and to send it back to the web page.
EDIT 2
I'm getting closer thanks to Tom.
This is what I did:
Template.doctypesList.rendered = ->
content = this.data.content
console.log content
templateName = "template_#{this.data._id}"
Template.__define__(templateName, () -> content)
rendered = UI.renderWithData(eval("Template.#{templateName}"),{name:"nicolas"})
UI.insert(rendered, $("#content_" + this.data._id).get(0))
This works excepted the fact that the name is not injected into the template. UI.renderWithData renders the template but without the data context...
The thing your are missing is the call to (undocumented!) Template.__define__ which requires the template name (pick something unique and clever) as the first argument and the render function which you get from your space bars compiler. When it is done you can use {{> UI.dynamic}} as #Slava suggested.
There is also another way to do it, by using UI.Component API, but I guess it's pretty unstable at the moment, so maybe I will skip this, at least for now.
Use UI.dynamic: https://www.discovermeteor.com/blog/blaze-dynamic-template-includes/
It is fairly new and didn't make its way to docs for some reason.
There are few ways to achieve what you want, but I would do it like this:
You're probably already using underscore.js, if not Meteor has core package for it.
You could use underscore templates (http://underscorejs.org/#template) like this:
var templateString = 'Dear <%= firstname %>'
and later compile it using
_.template(templateString, {firstname: "Tom"})
to get Dear Tom.
Of course you can store templateString in MongoDB in the meantime.
You can set delimiters to whatever you want, <%= %> is just the default.
Compiled template is essentially htmljs notation Meteor uses (or so I suppose) and it uses Template.template_name.lookup to render correct data. Check in console if Template.template_name.lookup("data_helper")() returns the correct data.
I recently had to solve this exact (or similar) problem of compiling templates client side. You need to make sure the order of things is like this:
Compiled template is present on client
Template data is present (verify with Template.template_name.lookup("data_name")() )
Render the template on page now
To compile the template, as #apendua have suggested, use (this is how I use it and it works for me)
Template.__define__(name, eval(Spacebars.compile(
newHtml, {
isTemplate: true,
sourceName: 'Template "' + name + '"'
}
)));
After this you need to make sure the data you want to render in template is available before you actually render the template on page. This is what I use for rendering template on page:
UI.DomRange.insert(UI.render(Template.template_name).dom, document.body);
Although my use case for rendering templates client side is somewhat different (my task was to live update the changed template overriding meteor's hot code push), but this worked best among different methods of rendering the template.
You can check my very early stage package which does this here: https://github.com/channikhabra/meteor-live-update/blob/master/js/live-update.js
I am fairly new to real-world programming so my code might be ugly, but may be it'll give you some pointers to solve your problem. (If you find me doing something stupid in there, or see something which is better done some other way, please feel free to drop a comment. That's the only way I get feedback for improvement as I am new and essentially code alone sitting in my dark corner).
Background: I downloaded a *.sql backup of my WordPress site's database, and replaced all instances of the old database table prefix with a new one (e.g. from the default wp_ to something like asdfghjkl_).
I've just learnt that WordPress uses serialized PHP strings in the database, and what I did will have messed with the integrity of the serialized string lengths.
The thing is, I deleted the backup file just before I learnt about this (as my website was still functioning fine), and installed a number of plugins since. So, there's no way I can revert back, and I therefore would like to know two things:
How can I fix this, if at all possible?
What kind of problems could this cause?
(This article states that, a WordPress blog for instance, could lose its settings and widgets. But this doesn't seem to have happened to me as all the settings for my blog are still intact. But I have no clue as to what could be broken on the inside, or what issues it'd pose in the future. Hence this question.)
Visit this page: http://unserialize.onlinephpfunctions.com/
On that page you should see this sample serialized string: a:1:{s:4:"Test";s:17:"unserialize here!";}. Take a piece of it-- s:4:"Test";. That means "string", 4 characters, then the actual string. I am pretty sure that what you did caused the numeric character count to be out of sync with the string. Play with the tool on the site mentioned above and you will see that you get an error if you change "Test" to "Tes", for example.
What you need to do is get those character counts to match your new string. If you haven't corrupted any of the other encoding-- removed a colon or something-- that should fix the problem.
I came to this same problem after trying to change the domain from localhost to the real URL. After some searching I found the answer in Wordpress documentation:
https://codex.wordpress.org/Moving_WordPress
I will quote what is written there:
To avoid that serialization issue, you have three options:
Use the Better Search Replace or Velvet Blues Update URLs plugins if you can > access your Dashboard.
Use WP-CLI's search-replace if your hosting provider (or you) have installed WP-CLI.
Run a search and replace query manually on your database. Note: Only perform a search and replace on the wp_posts table.
I ended up using WP-CLI which is able to replace things in the database without breaking serialization: http://wp-cli.org/commands/search-replace/
I know this is an old question, but better late than never, I suppose. I ran into this problem recently, after inheriting a database that had had a find/replace executed on serialized data. After many hours of researching, I discovered that this was because the string counts were off. Unfortunately, there was so much data with lots of escaping and newlines and I didn't know how to count in some cases and I had so much data that I needed something automated.
Along the way, I stumbled across this question and Benubird's post helped put me on the right path. His example code did not work in production use on complex data, containing numerous special characters and HTML, with very deep levels of nesting, and it did not properly handle certain escaped characters and encoding. So I modified it a bit and spent countless hours working through additional bugs to get my version to "fix" the serialized data.
// do some DB query here
while($res = db_fetch($qry)){
$str = $res->data;
$sCount=1; // don't try to count manually, which can be inaccurate; let serialize do its thing
$newstring = unserialize($str);
if(!$newstring) {
preg_match_all('/s:([0-9]+):"(.*?)"(?=;)/su',$str,$m);
# preg_match_all("/s:([0-9]+):(\"[^\"\\\\]*(?:\\\\.[^\"\\\\]*)*\")(?=;)/u",$str,$m); // alternate: almost works but leave quotes in $m[2] output
# print_r($m); exit;
foreach($m[1] as $k => $len) {
/*** Possibly specific to my case: Spyropress Builder in WordPress ***/
$m_clean = str_replace('\"','"',$m[2][$k]); // convert escaped double quotes so that HTML will render properly
// if newline is present, it will output directly in the HTML
// nl2br won't work here (must find literally; not with double quotes!)
$m_clean = str_replace('\n', '<br />', $m_clean);
$m_clean = nl2br($m_clean); // but we DO need to convert actual newlines also
/*********************************************************************/
if($sCount){
$m_new = $m[0][$k].';'; // we must account for the missing semi-colon not captured in regex!
// NOTE: If we don't flush the buffers, things like <img src="http://whatever" can be replaced with <img src="//whatever" and break the serialize count!!!
ob_end_flush(); // not sure why this is necessary but cost me 5 hours!!
$m_ser = serialize($m_clean);
if($m_new != $m_ser) {
print "Replacing: $m_new\n";
print "With: $m_ser\n";
$str = str_replace($m_new, $m_ser, $str);
}
}
else{
$m_len = (strlen($m[2][$k]) - substr_count($m[2][$k],'\n'));
if($len != $m_len) {
$newstr='s:'.$m_len.':"'.$m[2][$k].'"';
echo "Replacing: {$m[0][$k]}\n";
echo "With: $newstr\n\n";
$str = str_replace($m_new, $newstr, $str);
}
}
}
print_r($str); // this is your FIXED serialized data!! Yay!
}
}
A little geeky explanation on my changes:
I found that trying to count with Benubird's code as a base was too inaccurate for large datasets, so I ended up just using serialize to be sure the count was accurate.
I avoided the try/catch because, in my case, the try would succeed but just returned an empty string. So, I check for empty data instead.
I tried numerous regex's but only a mod on Benubird's would accurately handle all cases. Specifically, I had to modify the part that checked for the ";" because it would match on CSS like "width:100%; height:25px;" and broke the output. So, I used a positive lookahead to only match when the ";" was outside of the set of double quotes.
My case had lots of newlines, HTML, and escaped double quotes, so I had to add a block to clean that up.
There were a couple of weird situations where data would be replaced incorrectly by the regex and then the serialize would count it incorrectly as well. I found NOTHING on any sites to help with this and finally thought it might be related to caching or something like that and tried flushing the output buffer (ob_end_flush()), which worked, thank goodness!
Hope this helps someone... Took me almost 20 hours including the research and dealing with weird issues! :)
This script (https://interconnectit.com/products/search-and-replace-for-wordpress-databases/) can help to update an sql database with proper URLs everywhere, without encountering serialized data issues, because it will update the "characters count" that could throw your URLs out of sync whenever serialized data occurs.
The steps would be:
if you already have imported a messed up database (widgets not
working, theme options not there, etc), just drop that database
using PhpMyAdmin. That is, remove everything on it. Then export and
have at hand an un-edited dump of the old database.
Now you have to import the (un-edited) old database into the
newly created one. You can do this via an import, or copying over
the db from PhpMyAdmin. Notice that so far, we haven't done any
search and replace yet; we just have an old database content and
structure into a new database with its own user and password. Your site will be probably unaccessible at this point.
Make sure you have your WordPress files freshly uploaded to the
proper folder on the server, and edit your wp-config.php to make it
connect with the new database.
Upload the script into a "secret" folder - just for security
reasons - at the same level than wp-admin, wp-content, and wp-includes. Do not forget to remove it all once the search and
replace have taken place, because you risk to offer your DB details
open to the whole internet.
Now point your browser to the secret folder, and use the script's fine
interface. It is very self-explanatory. Once used, we proceed to
completely remove it from the server.
This should have your database properly updated, without any serialized data issues around: the new URL will be set everywhere, and serialized data characters counts will be accordingly updated.
Widgets will be passed over, and theme settings as well - two of the typical places that use serialized data in WordPress.
Done and tested solution!
If the error is due to the length of the strings being incorrect (something I have seen frequently), then you should be able to adapt this script to fix it:
foreach($strings as $key => $str)
{
try {
unserialize($str);
} catch(exception $e) {
preg_match_all('#s:([0-9]+):"([^;]+)"#',$str,$m);
foreach($m[1] as $k => $len) {
if($len != strlen($m[2][$k])) {
$newstr='s:'.strlen($m[2][$k]).':"'.$m[2][$k].'"';
echo "len mismatch: {$m[0][$k]}\n";
echo "should be: $newstr\n\n";
$strings[$key] = str_replace($m[0][$k], $newstr, $str);
}
}
}
}
I personally don't like working in PHP, or placing my DB credentials in an public file. I created a ruby script to fix serializations that you can run locally:
https://github.com/wsizoo/wordpress-fix-serialization
Context Edit:
I approached fixing serialization by first identifying serialization via regex, and then recalculating the byte size of the contained data string.
$content_to_fix.gsub!(/s:([0-9]+):\"((.|\n)*?)\";/) {"s:#{$2.bytesize}:\"#{$2}\";"}
I then update the specified data via an escaped sql update query.
escaped_fix_content = client.escape($fixed_content)
query = client.query("UPDATE #{$table} SET #{$column} = '#{escaped_fix_content}' WHERE #{$column_identifier} LIKE '#{$column_identifier_value}'")