While writing unit tests for my beam pipeline using PAssert, the pipeline outputs objects fine but the test fails during comparison with following assertion error:
java.lang.AssertionError: Decode pubsub message/ParMultiDo(DecodePubSubMessage).output:
Expected: iterable with items [<PubsubMessage{message=[123, 34, 104...], attributes={messageId=2be485e4-3e53-4468-a482-a49842b87ed5, dataPipelineId=bc957aa3-17e7-46d6-bc73-0924fa5674fa, region=us-west1, ingestionTimestamp=2020-02-02T12:34:56.789Z}, messageId=null}>] in any order
but: not matched: <PubsubMessage{message=[123, 34, 104...], attributes={messageId=2be485e4-3e53-4468-a482-a49842b87ed5, dataPipelineId=bc957aa3-17e7-46d6-bc73-0924fa5674fa, region=us-west1, ingestionTimestamp=2020-02-02T12:34:56.789Z}, messageId=null}>
I also tried encapsulating expectedOutputPubSubMessage in a list (apparently original output is in an Array) to no avail. All the given PAssert examples in documentation do a simple string or keyvalue comparison.
#RunWith(PowerMockRunner.class)
public class DataDecodePipelineTest implements Serializable {
#Rule
public TestPipeline p = TestPipeline.create();
#Test
public void testPipeline(){
PubsubMessage inputPubSubMessage =
new PubsubMessage(
TEST_ENCODED_PAYLOAD.getBytes(),
new HashMap<String, String>() {
{
put(MESSAGE_ID_NAME, TEST_MESSAGE_ID);
put(DATA_PIPELINE_ID_NAME, TEST_DATA_PIPELINE_ID);
put(INGESTION_TIMESTAMP_NAME, TEST_INGESTION_TIMESTAMP);
put(REGION_NAME, TEST_REGION);
}
});
PubsubMessage expectedOutputPubSubMessage =
new PubsubMessage(
TEST_DECODED_PAYLOAD.getBytes(),
new HashMap<String, String>() {
{
put(MESSAGE_ID_NAME, TEST_MESSAGE_ID);
put(DATA_PIPELINE_ID_NAME, TEST_DATA_PIPELINE_ID);
put(INGESTION_TIMESTAMP_NAME, TEST_INGESTION_TIMESTAMP);
put(REGION_NAME, TEST_REGION);
}
});
PCollection<PubsubMessage> input =
p.apply(Create.of(Collections.singletonList(inputPubSubMessage)));
PCollection<PubsubMessage> output =
input.apply("Decode pubsub message",
ParDo.of(new DataDecodePipeline.DecodePubSubMessage()));
PAssert.that(output).containsInAnyOrder(expectedOutputPubSubMessage);
p.run().waitUntilFinish();
}
}
Apparently, someone faced the exact same issue years ago which remains unresolved. Test pipeline comparing objects using PAssert containsInAnyOrder()
Just pass the expectedOutputPubSubMessage inside an array:
PAssert
.that(output)
.containsInAnyOrder(new PubsubMessage[] { expectedOutputPubSubMessage });
The problem is that you are comparing different objects
the return of your pipeline is a PCollection and you are comparing it against PubsubMessage
you have to create a PCollection from the expectedOutputPubSubMessage
Try this:
PAssert.that(output).containsInAnyOrder(Create.of(Collections.singletonList(expectedOutputPubSubMessage));
example: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/test/java/com/google/cloud/teleport/templates/PubsubToPubsubTest.java
I want to use the values of all properties that have some annotation. For the most part my code works, I get all the properties and only take those that have that annotation.
private inline fun <reified A : Annotation> (target: Any) {
target::class.memberProperties
.filter { it.annotations.any { annotation -> annotation is A } }
.forEach {
// How do I get the value here?
}
}
I wanted to use it.get(...) but get expects a Nothing as parameter. Same goes for getter. Calling it.call(target) does work but it looks wrong since there is an actuall get which I don't know how to invoke.
So what is the correct way of getting the properties value?
The problem boils down to the fact that T::class gives you a KClass<T>, whereas t::class gives you a KClass<out T>. Consider the following:
class Foo {
val foo = 2
val bar = 3
}
fun f() {
val any: Any = Foo()
any::class.memberProperties.forEach {
println(it.get(2)) // Oops
}
}
This would essentially attempt to access 2.foo and 2.bar, but it's not allowed because get errs on the side of caution instead of allowing a parameter of type Any. It appears doing t.javaClass.kotlin will produce a KClass<T>, however. Misusing it as above causes an IllegalArgumentException.
You can give the compiler some more help by providing a compile-time guarantee that the KClass will be for the type and nothing else:
private inline fun <reified A : Annotation, reified T : Any> foo(target: T) {
T::class.memberProperties
.filter { it.annotations.any { annotation -> annotation is A } }
.forEach {
println(it.get(target))
}
}
Unfortunately, I don't know if it's possible to specify A while deducing T from target. I haven't found a way past calling it like foo<Attr, Bar>(bar).
Alternatively, you can go through javaClass, though I'd wager it's less portable:
private inline fun <reified A : Annotation> foo(target: Any) {
target.javaClass.kotlin.memberProperties
.filter { it.annotations.any { annotation -> annotation is A } }
.forEach {
println(it.get(target))
}
}
We know this won't run into the above problem because we pass the same object in both cases. This also looks nicer on the caller's side, which might be worth the portability hit, assuming there is one.
I just got into Google Guava and it seems like a powerful tool and I see how you can use Predicates and filter by a specific property. How you can also chain predicates in FluentIterable My question is what's the best way to filter for a single property.
For example, if I have a collection of Cars. How do I filter the Cars.getPaintColor() to give me cars that are in Black, Red, and Yellow? Creating 3 separate predicates and using FluentIterable seems clumsy. Especially in my use, I could want possibly 10+ filters on the same property and I wouldn't want to create 10 Predicates.
Thanks you!
List<String> colorList = (List<String>)filterCriteria.get("Color");
List<String> makeList = (List<String>)filterCriteria.get("Make");
List<String> rimSizeList = (List<String>)filterCriteria.get("RimSize");
Predicate<String> predColor = Predicates.in(ImmutableSet.copyOf(colorList));
Predicate<CarObj> predDirection2 = Predicates.compose(predColor ,[????] );
Predicate<String> predMakeList = Predicates.in(ImmutableSet.copyOf(makeList));
Predicate<CarObj> predMakeList2 = Predicates.compose(predMakeList, [????] );
Predicate<String> predRimSize = Predicates.in(ImmutableSet.copyOf(rimSizeList));
Predicate<CarObj> predRimSize2 = Predicates.compose(predRimSize, [????] );
Collection<CarObj> filtered = FluentIterable.from(mAllCars)
.filter(predDirection2)
.filter(predMakeList2)
.filter(predRimSize2)
.toList();
Since I am using an List, I used copyOf instead of of when creating ImmutableSet.
I am not sure what to put in the second parameter of the compose. I am guessing it is something like this... in the CarObj class.
static Predicate<CarObj> byColor= new Predicate<CarObj>() {
public boolean apply(CarObj input) {
// What do I put here?
}
};
So, to check if a paint color is one of black, read or yellow, you'd want to create a Predicate that checks if a set contains that color:
Predicate<PaintColor> p = Predicates.in(ImmutableSet.of(
PaintColor.RED, PaintColor.BLACK, PaintColor.YELLOW));
You could then compose that with a Function<Car, PaintColor> that returns the paint color property of your class:
Predicate<Car> p2 = Predicates.compose(p, Car.GET_PAINT_COLOR_FUNCTION);
Edit:
By Car.GET_PAINT_COLOR_FUNCTION I just mean something like this:
public static final Function<Car, PaintColor> GET_PAINT_COLOR_FUNCTION =
new Function<Car, PaintColor>() {
#Override public PaintColor apply(Car car) {
return car.getPaintColor();
}
});
As I said in the comments, you can adapt that to your actual types as needed. For example, make it a Function<CarObj, String> instead.
The alternative to composing your extracting Function<Car, PaintColor> with a Predicates.in() as suggested by ColinD is to write your parameterized Predicate<Car>:
public class CarPaintColorPredicate implements Predicate<Car> {
private final PaintColor paintColor;
public CarPaintColorPredicate(PaintColor paintColor) {
this.paintColor = paintColor;
}
#Override
public boolean apply(#Nullable Car input) {
return input != null && input.getPaintColor() == paintColor;
}
}
which you can then use directly:
FluentIterable.from(cars)
.filter(new CarPaintColorPredicate(PaintColor.RED))
.toList();
or combine for multiple colors:
FluentIterable.from(cars)
.filter(Predicates.or(
new CarPaintColorPredicate(PaintColor.RED),
new CarPaintColorPredicate(PaintColor.BLACK)))
.toList();
or even combine with other types of predicates:
FluentIterable.from(cars)
.filter(new CarPaintColorPredicate(PaintColor.RED))
.filter(new CarMakePredicate("Ferrari"))
.toList();
To be complete, the version with the Function<Car, PaintColor> is as follows:
public enum CarPaintColorFunction implements Function<Car, PaintColor> {
INSTANCE;
#Override
public PaintColor apply(#Nullable Car input) {
return input == null ? null : input.getPaintColor();
}
}
The Function simply returns the value of the property, which is then compared to the collection (hopefully a Set) of accepted values through the Predicate composition:
FluentIterable.from(cars)
.filter(Predicates.compose(
Predicates.in(Sets.immutableEnumSet(PaintColor.RED, PaintColor.BLACK)),
CarPaintColorFunction.INSTANCE))
.toList();
All that's really explained in the Functional Explained page in the Guava Wiki.
I have a variable like this:
People model=Mock()//created by spock' mock
How to change the variable name "model" to "model_0" by reflection?
I doubt there is any way to do that with reflection. I'm with #tim_yates in this one.
I'm no bytecode/compiler expert, but i do believe variable's names turn into symbols upon compilation, so they are not changeable at all. The following Java class:
public class Var {
void a() {
int myVar = 1;
myVar += 1;
}
}
Upon compilation and decompilation (using jd-gui), the code turns into:
public class Var
{
void a()
{
int i = 1;
i++;
}
}
The variable name changed.
On Groovy, you can go for ASTs, which will give you full power over the generated tree. The following class:
class Asts {
def wat() {
Integer myVar = 90
}
}
Will generate the following AST:
Now you can write your own AST transformation to make the changes. Doesn't seem worth to me, though, a collection or map should suffice.
I have BigDecimal objects serialized with BlazeDS to Actionscript. Once they hit Actionscript as Number objects, they have values like:
140475.32 turns into 140475.31999999999998
How do I deal with this? The problem is that if I use a NumberFormatter with precision of 2, then the value is truncated to 140475.31. Any ideas?
This is my generic solution for the problem (I have blogged about this here):
var toFixed:Function = function(number:Number, factor:int) {
return Math.round(number * factor)/factor;
}
For example:
trace(toFixed(0.12345678, 10)); //0.1
Multiply 0.12345678 by 10; that gives us 1.2345678.
When we round 1.2345678, we get 1.0,
and finally, 1.0 divided by 10 equals 0.1.
Another example:
trace(toFixed(1.7302394309234435, 10000)); //1.7302
Multiply 1.7302394309234435 by 10000; that gives us 17302.394309234435.
When we round 17302.394309234435 we get 17302,
and finally, 17302 divided by 10000 equals 1.7302.
Edit
Based on the anonymous answer below, there is a nice simplification for the parameter on the method that makes the precision much more intuitive. e.g:
var setPrecision:Function = function(number:Number, precision:int) {
precision = Math.pow(10, precision);
return Math.round(number * precision)/precision;
}
var number:Number = 10.98813311;
trace(setPrecision(number,1)); //Result is 10.9
trace(setPrecision(number,2)); //Result is 10.98
trace(setPrecision(number,3)); //Result is 10.988 and so on
N.B. I added this here just in case anyone sees this as the answer and doesn't scroll down...
Just a slight variation on Frasers Function, for anyone who is interested.
function setPrecision(number:Number, precision:int) {
precision = Math.pow(10, precision);
return (Math.round(number * precision)/precision);
}
So to use:
var number:Number = 10.98813311;
trace(setPrecision(number,1)); //Result is 10.9
trace(setPrecision(number,2)); //Result is 10.98
trace(setPrecision(number,3)); //Result is 10.988 and so on
i've used Number.toFixed(precision) in ActionScript 3 to do this: http://livedocs.adobe.com/flex/3/langref/Number.html#toFixed%28%29
it handles rounding properly and specifies the number of digits after the decimal to display - unlike Number.toPrecision() that limits the total number of digits to display regardless of the position of the decimal.
var roundDown:Number = 1.434;
// will print 1.43
trace(roundDown.toFixed(2));
var roundUp:Number = 1.436;
// will print 1.44
trace(roundUp.toFixed(2));
I converted the Java of BigDecimal to ActionScript.
We had no choices since we compute for financial application.
http://code.google.com/p/bigdecimal/
You can use property: rounding = "nearest"
In NumberFormatter, rounding have 4 values which you can choice: rounding="none|up|down|nearest". I think with your situation, you can chose rounding = "nearest".
-- chary --
I discovered that BlazeDS supports serializing Java BigDecimal objects to ActionScript Strings as well. So if you don't need the ActionScript data to be Numbers (you are not doing any math on the Flex / ActionScript side) then the String mapping works well (no rounding weirdness). See this link for the BlazeDS mapping options: http://livedocs.adobe.com/blazeds/1/blazeds_devguide/help.html?content=serialize_data_2.html
GraniteDS 2.2 has BigDecimal, BigInteger and Long implementations in ActionScript3, serialization options between Java / Flex for these types, and even code generation tools options in order to generate AS3 big numbers variables for the corresponding Java ones.
See more here: http://www.graniteds.org/confluence/display/DOC22/2.+Big+Number+Implementations.
guys, just check the solution:
protected function button1_clickHandler(event:MouseEvent):void
{
var formatter:NumberFormatter = new NumberFormatter();
formatter.precision = 2;
formatter.rounding = NumberBaseRoundType.NEAREST;
var a:Number = 14.31999999999998;
trace(formatter.format(a)); //14.32
}
I ported the IBM ICU implementation of BigDecimal for the Actionscript client. Someone else has published their nearly identical version here as a google code project. Our version adds some convenience methods for doing comparisons.
You can extend the Blaze AMF endpoint to add serialization support for BigDecimal. Please note that the code in the other answer seems incomplete, and in our experience it fails to work in production.
AMF3 assumes that duplicate objects, traits and strings are sent by reference. The object reference tables need to be kept in sync while serializing, or the client will loose sync of these tables during deserialization and start throwing class cast errors, or corrupting the data in fields that don't match, but cast ok...
Here is the corrected code:
public void writeObject(final Object o) throws IOException {
if (o instanceof BigDecimal) {
write(kObjectType);
if(!byReference(o)){ // if not previously sent
String s = ((BigDecimal)o).toString();
TraitsInfo ti = new TraitsInfo("java.math.BigDecimal",false,true,0);
writeObjectTraits(ti); // will send traits by reference
writeUTF(s);
writeObjectEnd(); // for your AmfTrace to be correctly indented
}
} else {
super.writeObject(o);
}
}
There is another way to send a typed object, which does not require Externalizable on the client. The client will set the textValue property on the object instead:
TraitsInfo ti = new TraitsInfo("java.math.BigDecimal",false,false,1);
ti.addProperty("textValue");
writeObjectTraits(ti);
writeObjectProperty("textValue",s);
In either case, your Actionscript class will need this tag:
[RemoteClass(alias="java.math.BigDecimal")]
The Actionscript class also needs a text property to match the one you chose to send that will initialize the BigDecimal value, or in the case of the Externalizable object, a couple of methods like this:
public function writeExternal(output:IDataOutput):void {
output.writeUTF(this.toString());
}
public function readExternal(input:IDataInput):void {
var s:String = input.readUTF();
setValueFromString(s);
}
This code only concerns data going from server to client. To deserialize in the other direction from client to server, we chose to extend AbstractProxy, and use a wrapper class to temporarily store the string value of the BigDecimal before the actual object is created, due to the fact that you cannot instantiate a BigDecimal and then assign the value, as the design of Blaze/LCDS expects should be the case with all objects.
Here's the proxy object to circumvent the default handling:
public class BigNumberProxy extends AbstractProxy {
public BigNumberProxy() {
this(null);
}
public BigNumberProxy(Object defaultInstance) {
super(defaultInstance);
this.setExternalizable(true);
if (defaultInstance != null)
alias = getClassName(defaultInstance);
}
protected String getClassName(Object instance) {
return((BigNumberWrapper)instance).getClassName();
}
public Object createInstance(String className) {
BigNumberWrapper w = new BigNumberWrapper();
w.setClassName(className);
return w;
}
public Object instanceComplete(Object instance) {
String desiredClassName = ((BigNumberWrapper)instance).getClassName();
if(desiredClassName.equals("java.math.BigDecimal"))
return new BigDecimal(((BigNumberWrapper)instance).stringValue);
return null;
}
public String getAlias(Object instance) {
return((BigNumberWrapper)instance).getClassName();
}
}
This statement will have to execute somewhere in your application, to tie the proxy object to the class you want to control. We use a static method:
PropertyProxyRegistry.getRegistry().register(
java.math.BigDecimal.class, new BigNumberProxy());
Our wrapper class looks like this:
public class BigNumberWrapper implements Externalizable {
String stringValue;
String className;
public void readExternal(ObjectInput arg0) throws IOException, ClassNotFoundException {
stringValue = arg0.readUTF();
}
public void writeExternal(ObjectOutput arg0) throws IOException {
arg0.writeUTF(stringValue);
}
public String getStringValue() {
return stringValue;
}
public void setStringValue(String stringValue) {
this.stringValue = stringValue;
}
public String getClassName() {
return className;
}
public void setClassName(String className) {
this.className = className;
}
}
We were able to reuse one of the available BigDecimal.as classes on the web and extended blazeds by sublassing from AMF3Output, you'll need to specify your own endpoint class in the flex xml files, in that custom endpoint you can insert your own serializer that instantiates an AMF3Output subclass.
public class EnhancedAMF3Output extends Amf3Output {
public EnhancedAMF3Output(final SerializationContext context) {
super(context);
}
public void writeObject(final Object o) throws IOException {
if (o instanceof BigDecimal) {
write(kObjectType);
writeUInt29(7); // write U290-traits-ext (first 3 bits set)
writeStringWithoutType("java.math.BigDecimal");
writeAMFString(((BigDecimal)o).toString());
} else {
super.writeObject(o);
}
}
}
as simple as that! then you have native BigDecimal support using blazeds, wooohoo!
Make sure your BigDecimal as3 class implements IExternalizable
cheers, jb
Surprisingly the round function in MS Excel gives us different values then you have presented above.
For example in Excel
Round(143,355;2) = 143,36
So my workaround for Excel round is like:
public function setPrecision(number:Number, precision:int):Number {
precision = Math.pow(10, precision);
const excelFactor : Number = 0.00000001;
number += excelFactor;
return (Math.round(number * precision)/precision);
}
If you know the precision you need beforehand, you could store the numbers scaled so that the smallest amount you need is a whole value. For example, store the numbers as cents rather than dollars.
If that's not an option, how about something like this:
function printTwoDecimals(x)
{
printWithNoDecimals(x);
print(".");
var scaled = Math.round(x * 100);
printWithNoDecimals(scaled % 100);
}
(With however you print with no decimals stuck in there.)
This won't work for really big numbers, though, because you can still lose precision.
You may vote and watch the enhancement request in the Flash PLayer Jira bug tracking system at https://bugs.adobe.com/jira/browse/FP-3315
And meanwhile use the Number.toFixed() work-around see :
(http://livedocs.adobe.com/flex/3/langref/Number.html#toFixed%28%29)
or use the open source implementations out there : (http://code.google.com/p/bigdecimal/) or (http://www.fxcomps.com/money.html)
As for the serialization efforts, well, it will be small if you use Blazeds or LCDS as they do support Java BigDecimal serialization (to String) cf. (http://livedocs.adobe.com/livecycle/es/sdkHelp/programmer/lcds/wwhelp/wwhimpl/common/html/wwhelp.htm?context=LiveDocs_Parts&file=serialize_data_3.html)
It seems more like a transport problem, the number being correct but the scale ignored. If the number has to be stored as a BigDecimal on the server you may want to convert it server side to a less ambiguous format (Number, Double, Float) before sending it.