add or create new attributes using NCO - netcdf

I have one .nc file in which I want to create a new attribute
like
float Longitude_Aerosol_NearUV_Swath(nTimes_Aerosol_NearUV_Swath, nXtrack_Aerosol_NearUV_Swath) ;
Longitude_Aerosol_NearUV_Swath:standard_name = "longitude" ;
Longitude_Aerosol_NearUV_Swath:long_name = "longitude" ;
Longitude_Aerosol_NearUV_Swath:units = "degrees_east" ;
Longitude_Aerosol_NearUV_Swath:_CoordinateAxisType = "Lon" ;
please let me know how to add this in the file.
Thanks


Read the fine manual here
ncatted -a attribute_name,variable_name,o,c,'text string' in.nc out.nc

Related

How to get the changed files list for a commit with JGit

I want to get the changed files between two commits, not the diff information, how can I use JGit to make it?

With two refs pointing at the two commits it should suffice to do the following to iterate all changes between the commits:
ObjectId oldHead = repository.resolve("HEAD^^^^{tree}");
ObjectId head = repository.resolve("HEAD^{tree}");
// prepare the two iterators to compute the diff between
try (ObjectReader reader = repository.newObjectReader()) {
CanonicalTreeParser oldTreeIter = new CanonicalTreeParser();
oldTreeIter.reset(reader, oldHead);
CanonicalTreeParser newTreeIter = new CanonicalTreeParser();
newTreeIter.reset(reader, head);
// finally get the list of changed files
try (Git git = new Git(repository)) {
List<DiffEntry> diffs= git.diff()
.setNewTree(newTreeIter)
.setOldTree(oldTreeIter)
.call();
for (DiffEntry entry : diffs) {
System.out.println("Entry: " + entry);
}
}
}
}
There is a ready-to-run example snippet contained in the jgit-cookbook

How to make URI locations from AST to map on a file read

In ClaiR it is not (yet) possible to write changes made in the AST back to file.
For this reason, I create a list lrel[int, int, str] changes = []; with startposition and endposition of the substring to remove, and a string with which it needs to be replaced.
When I have a full list of changes I want to make to a source file, I sort the changes and open the file with fb = chars(readFile(f));
make the changes
public list[int] changeCharList(list[int] charList, lrel[int, int, str] changesList) {
int offset = 0;
for (t <- [0 .. size(changesList)]) {
tuple[int startIndex, int endIndex, str changeWithString] change = changesList[t];
int startIndexWithOffset = change.startIndex + offset;
int endIndexWithOffset = change.endIndex + offset;
list[int] changeWithChars = chars(change.changeWithString);
for (i <- [startIndexWithOffset .. endIndexWithOffset]) {
charList = delete(charList, startIndexWithOffset);
}
for (i <- [0 .. size(changeWithChars)]) {
charList = insertAt(charList, startIndexWithOffset + i, changeWithChars[i]);
}
offset += size(changeWithChars) - (change.endIndex - change.startIndex);
}
return charList;
}
and write to file writeFileBytes(f, fb);
This approach works for source files without expanded macros, but it does not work for sources files with expanded macros. In the later case the offsets used in the AST do not map the offsets with the file opened using readFile.
As a workaround I can comment macros before running Rascal and uncomment them after running Rascal. I do not like this.
Is there a way to recalculate the offsets in such a way that the AST offsets map the file read offsets?

ffmpeg command to create thumbnail sprites

To output a single frame from ffmpeg I can do:
ffmpeg -i input.flv -ss 00:00:14.435 -vframes 1 out.png
And to output an image every second, I can do:
ffmpeg -i input.flv -vf fps=1 out%d.png
Would there be a way to create a thumbnail sprite from these outputs to help with the creation of a vtt file for thumbs on seek position?

Here is an example to create jpeg files (280x180) from mp4 video, using ffmpeg, and then assembling this thumbnails into a sprite (png format) using PHP gd2 + writing a VTT file for the video player.
First create 1 image per second with ffmpeg:
ffmpeg -i sprite/MyVideoFile.mp4 -r 1 -s 280x180 -f image2 sprite/thumbs/thumb-%%d.jpg
Then create sprite file + vtt file (example with PHP):
$dirToScan = 'thumbs/';
$filePrefix = 'thumb-';
$fileSuffix = '.jpg';
$thumbWidth = 280;
$thumbHeight = 180;
$imageFiles = array();
$spriteFile = 'sprite.png';
$imageLine = 20;
$vttFile = 'sprite.vtt';
$dst_x = 0;
$dst_y = 0;
# read the directory with thumbnails, file name in array
foreach (glob($dirToScan.$filePrefix.'*'.$fileSuffix) as $filename) {
array_push($imageFiles,$filename);
}
natsort($imageFiles);
#calculate dimension for the sprite
$spriteWidth = $thumbWidth*$imageLine;
$spriteHeight = $thumbHeight*(floor(count($imageFiles)/$imageLine)+1);
# create png file for sprite
$png = imagecreatetruecolor($spriteWidth,$spriteHeight);
# open vtt file
$handle = fopen($vttFile,'wb+');
fwrite($handle,'WEBVTT'."\n");
# insert thumbs in sprite and write the vtt file
foreach($imageFiles AS $file) {
$counter = str_replace($filePrefix,'',str_replace($fileSuffix,'',str_replace($dirToScan,'',$file)));
$imageSrc = imagecreatefromjpeg($file);
imagecopyresized ($png, $imageSrc, $dst_x , $dst_y , 0, 0, $thumbWidth, $thumbHeight, $thumbWidth,$thumbHeight);
$varTCstart = gmdate("H:i:s", $counter-1).'.000';
$varTCend = gmdate("H:i:s", $counter).'.000';
$varSprite = $spriteFile.'#xywh='.$dst_x.','.$dst_y.','.$thumbWidth.','.$thumbHeight;
fwrite($handle,$counter."\n".$varTCstart.' --> '.$varTCend."\n".$varSprite."\n\n");
create new line after 20 images
if ($counter % $imageLine == 0) {
$dst_x=0;
$dst_y+=$thumbHeight;
}
else {
$dst_x+=$thumbWidth;
}
}
imagepng($png,$spriteFile);
fclose($handle);
VTT file looks like:
WEBVTT
1
00:00:00.000 --> 00:00:01.000
sprite.png#xywh=0,0,280,180
2
00:00:01.000 --> 00:00:02.000
sprite.png#xywh=280,0,280,180
3
00:00:02.000 --> 00:00:03.000
sprite.png#xywh=560,0,280,180
...

I am not completely sure what you need/mean with sprite for a vtt file, but there is a nice tool that allows you to combine single images into a big overview picture:
ImageMagick comes with a handy tool called montage
montage - create a composite image by combining several separate images. The images are tiled on the composite image optionally adorned with a border, frame, image name, and more.
You can put the thumbnails together on one or more images with the following command:
montage *.png -tile 4x4 overview.png
It will automatically generate the number of needed pictures to give you an overview.

A little improvement proposal to the answer given by MR_1204: if the final sprite is a PNG, I would avoid the quality decrease caused by JPG compression and rather save the temporary thumbnails as PNGs:
ffmpeg -i sprite/MyVideoFile.mp4 -r 1 -s 280x180 -f image2 sprite/thumbs/thumb-%%d.png
Also, saving to PNG is usually a (tiny) little bit faster than saving to JPG.
Instead, to keep the download small, it might make sense to save (only) the final sprite image as JPG, which in case of PHP and GD only requires to replace imagepng(...) with imagejpeg(...).

Python solution of MR_1204's solution
import ffmpeg
import logging
from pathlib import Path
from config import temp
import os
from PIL import Image
from datetime import datetime, timedelta
import math
logger = logging.getLogger(__name__)
class ScreenshotExtractor:
def __init__(self):
self.screenshot_folder = f"{temp}ss"
self.gap = 10
self.size = "177x100"
self.col = 5
self.ss_width = 177
self.ss_height = 100
def create_sprite_sheet(self, name):
path, dirs, files = next(os.walk(self.screenshot_folder))
file_count = len(files)
img_width = self.ss_width * self.col
img_height = self.ss_height * math.ceil(file_count/self.col)
file_name = f"{name}.jpg"
out_image = Image.new('RGB', (img_width, img_height))
webvtt = "WEBVTT\n\n"
for count, img_file in enumerate(files):
img = Image.open(f"{self.screenshot_folder}/{img_file}")
st_time = timedelta(seconds=count * self.gap)
end_time = timedelta(seconds=(count + 1) * self.gap)
# Adding img to out_file
x_mod = int(count / self.col)
x = 0 + (count - (self.col * x_mod)) * self.ss_width
y = 0 + x_mod * self.ss_height
out_image.paste(img, (x, y))
sprite = f"{file_name}#xywh={x},{y},{self.ss_width},{self.ss_height}"
webvtt += f"{count + 1}\n0{str(st_time)}.000 --> 0{str(end_time)}.000\n{sprite}\n\n"
out_image.save(f"{temp}{file_name}", quality=90)
f = open(f"{temp}{name}.vtt", "w")
f.write(webvtt)
f.close()
return True
def extract_screenshots(self, file_uri, name):
try:
Path(self.screenshot_folder).mkdir(parents=True, exist_ok=True)
vod = ffmpeg.input(file_uri)
vod.output(f"{self.screenshot_folder}/{name}_%04d.png", r=1/self.gap, s=self.size).run()
return True
except Exception as e:
logger.warning(e)
return False
def run(self, file_uri, name):
# TODO - Actually do logic here
self.extract_screenshots(file_uri, name)
self.create_sprite_sheet(name)
Hope this helps somebody

Join lines based on a starting value using UNIX commands

Here I am again, with another UNIX requirement (as my knowledge in UNIX is limited to basic commands).
I have a file that looks like this (and has about 30 million lines)
123456789012,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
123456789012,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
123456789012,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
234567890123,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
234567890123,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
345678901234,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
345678901234,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
345678901234,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
456789012345,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
567890123456,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
567890123456,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
The final output should be like this (without the first value repeating in the joined portions)
123456789012,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
234567890123,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
345678901234,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
456789012345,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
567890123456,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
However, if the above output is a bit complicated, an output like below is also fine. Because I can load the file into Oracle11g and get rid of the redundant columns.
123456789012,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,123456789012,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,123456789012,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
234567890123,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,234567890123,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
345678901234,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,345678901234,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,345678901234,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
456789012345,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
567890123456,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,567890123456,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0

Using awk is sufficient; it is a control-break report of sorts. Since the lines with the same key are grouped together — a very important point — it is fairly simple.
awk -F, '{ if ($1 != saved)
{
if (saved != 0) print saved "," list
saved = $1
list = ""
}
pad = ""
for (i = 2; i <= NF; i++) { list = list pad $i; pad = "," }
}
END { if (saved != 0) print saved, list }'
You can feed the data as standard input or list the files to be processed after the final single quote.
Sample output:
123456789012,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
234567890123,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
345678901234,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
456789012345,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
567890123456 PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
The code uses saved to keep a track of the key column value that it is accumulating. When the key column changes, print out the saved values (if there are any) and reset for the new set of lines. At the end, print out the saved values (if there are any). The code deals with an empty file gracefully, therefore.
Perl options
#!/usr/bin/env perl
use strict;
use warnings;
my $saved = "";
my $list;
while (<>)
{
chomp;
my($key,$value) = ($_ =~ m/^([^,]+)(,.*)/);
if ($key ne $saved)
{
print "$saved$list\n" if $saved;
$saved = $key;
$list = "";
}
$list .= $value;
}
print "$saved$list\n" if $saved;
Or, if you really want to, you can saved writing the loop (and using strict and warnings) with:
perl -n -e 'chomp;
($key,$value) = ($_ =~ m/^([^,]+)(,.*)/);
if ($key ne $saved)
{
print "$saved$list\n" if $saved;
$saved = $key;
$list = "";
}
$list .= $value;
} END {
print "$saved$list\n" if $saved;'
That could be squished down to a single (rather long) line. The } END { is a piece of Perl weirdness; the -n option creates a loop while (<>) { … } and interpolates the script in the -e argument into it, so the } in } END { terminates that loop and then creates an END block which is ended by the } that Perl provided. Yes, documented and supported; yes, extremely weird (so I wouldn't do it; I'd use the Perl script shown first).

This awk script does what you want:
BEGIN { FS = OFS = "," }
NR == 1 { a[++n] = $1 }
a[1] != $1 { for(i=1; i<=n; ++i) printf "%s%s", a[i], (i<n?OFS:ORS); n = 1 }
{ a[1] = $1; for(i=2;i<=NF;++i) a[++n] = $i }
END { for(i=1; i<=n; ++i) printf "%s%s", a[i], (i<n?OFS:ORS) }
It stores all of the fields with the same first column in an array. When the first column differs, it prints out all of the elements of the array. Use it like awk -f join.awk file.
Output:
123456789012,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
234567890123,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
345678901234,PID=1,AID=2,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
456789012345,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0
567890123456,PID=2,AID=1,EQOSID=1,PDPTY=IPV4,PDPCH=2-0,PID=3,AID=8,EQOSID=1,PDPTY=IPV4,PDPCH=2-0

Here are some Python options, if you decide to go that route... First will work for multiple input files and non-sequential identical indices. Second doesn't read the whole file into memory.
(Note, I know it is not convention, but I intentionally use UpperCase for variables to make it clear what is a user-defined variable and what is a special python word.)
#! /usr/bin/env python
# -*- coding: utf-8 -*-
"""
concatenate comma-separated values based on first value
Usage:
catfile.py *.txt > output.dat
"""
import sys
if len(sys.argv)<2:
sys.stderr.write(__doc__)
else:
FileList = sys.argv[1:]
IndexList = []
OutDict = {}
for FileName in FileList:
with open(FileName,'rU') as FStream:
for Line in FStream:
if Line:
Ind,TheRest = Line.rstrip().split(",",1)
if Ind not in IndexList:
IndexList.append(Ind)
OutDict[Ind] = OutDict.get(Ind,"") + "," + TheRest
for Ind in IndexList:
print Ind + OutDict[Ind]
Here is a different version which doesn't load the whole file into memory, but requires that the identical Indices all occur in order, and it only runs on one file:
#! /usr/bin/env python
# -*- coding: utf-8 -*-
"""
concatenate comma-separated values based on first value
Usage:
catfile.py *.txt > output.dat
"""
import sys
if len(sys.argv)<2:
sys.stderr.write(__doc__)
else:
FileName = sys.argv[1]
OutString = ''
PrevInd = ''
FirstLine = True
with open(FileName,'rU') as FStream:
for Line in FStream:
if "," in Line:
Ind,TheRest = Line.rstrip().split(",",1)
if Ind != PrevInd:
if not FirstLine:
print PrevInd+OutString
PrevInd = Ind
OutString = TheRest
FirstLine = False
else:
OutString += ","+TheRest
print Ind + OutString
More generally, you can run these with by saving them as say catfile.py and then doing python catfile.py inputfile.txt > outputfile.txt. Or for longer term solutions, make a scripts directory, add it to your $PATH, make them executable with chmod u+x catfile.py and then you can just type the name of the script from any directory. But that is another topic that you would want to research.

A way without array:
BEGIN { FS = OFS = "," ; ORS = "" }
{
if (lid == $1) { $1 = "" ; print $0 }
else { print sep $0 ; lid = $1 ; sep = "\n" }
}
END { if (NR) print }
Note: if you don't need a newline at the end, remove the END block.

This might work for you (GNU sed):
sort file | sed -r ':a;$!N;s/^(([^,]*),.*)\n\2/\1/;ta;P;D'
Sort the file (if need be) and then delete newline and key where duplicates appear.

How to hide layer by id?

I am writing generator plugin and I get document structure using method generator.getDocumentInfo(). It returns document object containing layer objects in tree structure. document object has property document.id and each layer has property layer.id.
Goal: I want to hide layer - I know only document id and layer id.
Problem: The only method to hide layer in generator plugin I found is evaluateJSXString() method. This is fine but I don't know how to access document by id and layer by id. According http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/devnet/photoshop/pdfs/photoshop-cc-javascript-ref.pdf document has no id property and layer also has no id property. All I found is that app.documents is an array of documents (but index is not id) and app.document[i].layers is an array of layers but it contains only top level layers and each top level layer contains child layers.
The only option I see is to write JSX script which will first traverse app.documents array to find doc with for example matching file name and then it will search for a layer in document.layers tree structure..
Is there any other options?
How generator provides document and layer IDs when using generator.getDocumentInfo()? Is some generatpr-internal notation?

I see this is an old question had the same question. I was able to come up with a solution using generator's method evaluateJSXString. You can execute extendscript with evaluateJSXString with in your generator plugin. There is no looping involved here. Just by layerID.
Note: layerID is a variable that holds the layer ID and it is concatenated to the string to be evaluated.
To show the layer:
var changeVisibilityString = " var ref = new ActionReference(); \
ref.putIdentifier(charIDToTypeID('Lyr '), " + layerID + " ); \
var desc = new ActionDescriptor(); \
desc.putReference(charIDToTypeID('null'), ref); \
desc.putBoolean(charIDToTypeID('MkVs'), false); \
executeAction( charIDToTypeID('Shw '), desc);"
generator.evaluateJSXString(changeVisibilityString);
To hide the layer:
var changeVisibilityString = " var ref = new ActionReference(); \
ref.putIdentifier(charIDToTypeID('Lyr '), " + layerID + " ); \
var desc = new ActionDescriptor(); \
desc.putReference(charIDToTypeID('null'), ref); \
desc.putBoolean(charIDToTypeID('MkVs'), false); \
executeAction( charIDToTypeID('Hd '), desc);"
generator.evaluateJSXString(changeVisibilityString);

Resources