We have been using PubSubLite in our Go program without any issues and I just started using the Java library with Beam.
Using the PubSubLite IO, we get PCollection of SequencedMessage specifically: https://cloud.google.com/java/docs/reference/google-cloud-pubsublite/latest/com.google.cloud.pubsublite.proto.SequencedMessage
Now, from it I can get the data by doing something like:
message.getMessage().getData().toByteArray()
and then doing the normal conversion.
But for attributes, I cannot seem to get it correctly, just the value. In Go, I could do:
msg.Attributes["attrKey"]
but when I do:
message.getMessage().getAttributesMap().get("attrKey")
I am getting an Object which I cannot seem to convert to just string value of it. As far as I understand, it returns a Map<String, AttributeValues> and they all seem to be just wrapper over the internal protobuf. Also, Map is an interface so how do I get to the actual implementation to get the underlying value of each of the attribute.
The SequencedMessage attributes represent a multimap of string to bytes, not a map of string to string like in standard Pub/Sub. In the go client, by default the client will error if there are multiple values for a given key or if any of the values is not valid UTF-8, and thus presents a map[string]string interface.
When you call message.getMessage().getAttributesMap().get("attrKey"), you have a value of type AttributeValues which is a holder for a list of ByteStrings. To convert this to a single String, you would need to throw if the list is not of length 1, then call toStringUtf8 on the byte string element with index 0.
If you wish to interact with the standard Pub/Sub message format like you would in go, you can convert to this format by doing:
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
import org.apache.beam.sdk.io.gcp.pubsublite.CloudPubsubTransforms;
PCollection<SequencedMessage> messages = ...
PCollection<PubsubMessage> transformed = messages.apply(CloudPubsubTransforms.toCloudPubsubMessages());
Related
I need to log messages that will possibly exceed the 32k character limit. How can I extend the LogWriter with a debug(character) or what would be the best way to use the Logging Framework to log messages that are greater than 32k characters
There is a way, but it takes a bit of code.
The first step is to use the debug(LogMessage, ...) method, rather than debug(character, ...) method.
You will need to build the LogMessage but can't use its public properties (since they are also character. So you will need to use the LogMessage's AddContext method. This takes a character for the key and a Progress.Lang.Object as the value. You can use an instance of OpenEdge.Core.String, which holds longchar values.
using OpenEdge.Logging.*.
using OpenEdge.Core.*.
define variable logger as ILogWriter no-undo.
define variable logMsg as LogMessage no-undo.
logger = LoggerBuilder:GetLogger('something').
// do stuff
define variable longcharWithLotsOfData as longchar no-undo.
logMsg = new LogMessage(logger:Name, 'short message').
logMsg:AddContext('long-message', new String(longcharWithLotsOfData)).
logger:debug(logMsg).
You will probably also need to add your own filter to read this message from that context, and write it into the logfile. You can see an example of how to create one here .
You'll need to write to a 'named file' rather than the LOG-MANAGER since the WRITE-MESSAGE() method only takes a character for the message.
The writer will need to output the String object's Value property; ToString() returns a character. The filter will need to check the type of the object returned from the GetContext() method, and cast it to get the value.
You will need to use the COPY-LOB... APPEND statement to write the longchar value to the output file.
There are basically 3 steps
Create a class that implements ILoggerFilter
Add the filter definition to the filter property in logging.config
Add the filter to your logger(s) in logging.config
val args = Bundle()
args.putString("type", details.type)
navigator.navigate(context!!, findNavController(), Destination.TYPE, args)
I am quite confused as to why in the receiving fragment when I go to access the arguments I have passed through it is responding with...
val type: String = arguments.getString("type")
The arguments.getString is all underlined red and says "Required String Found String?" But how when I called method "putString"?!?
It is resulting in text not being rendered in the new fragment and I assume this is a nullability issue.
It's a matter of knowledge that is available in the receiving Fragment.
The Fragment is not aware of how its arguments were created (or modified) so it has to assume the "type" key you're looking for might not be in the arguments Bundle. That's why it returns a nullable (String?) result (the null value would mean absent in arguments).
Your fragment might be created in many places in your app and its arguments might have been modified in many places. We have no way of tracking that.
There are different solutions for this problem, depending on your approach in other parts of the code and how "confident" you are in creating of your Fragment.
I would usually choose a solution in which I assume setting the type is mandatory. Therefore if the type is absent - I fail fast. That would mean the Fragment was misused.
val type: String = arguments!!.getString("type")!!
The code above will crash if either:
a) arguments weren't set, or
b) String with type wasn't put in the arguments Bundle.
You are right, that is a : null ability issue.
First you should be sure if you are expecting a value, so try adding "?" or "!!", i would recommend "?", or go with the block of if {} else
To read the string safely you can use:
val type: String = arguments?.getString("type").orEmpty()
The orEmpty call at the end ensures that a valid String is returned even if either arguments or getString() returns null.
The method signature for getString() returns a nullable String. This is because at compile time, the compiler can't know if the value exists in the bundle or not. You will have the same issue when retrieving anything from any Map.
If you know for certain that the value in the bundle or map should exist at the time you call getString(), you can use the !! operator. That's what it's there for. When you know something should always be there, it is appropriate to want an exception to be thrown (in this case KNPE) if it's not there so you can easily find any programming error during testing.
isEmpty() or ?.let aren't helpful in this particular case because they would just be masking a programming error and making it harder to discover or debug.
Values that can be converted to a JSON string via json.dumps are:
Scalars: Numbers and strings
Containers: Mapping and Iterable
Union[str, int, float, Mapping, Iterable]
Do you have a better suggestion?
Long story short, you have the following options:
If you have zero idea how your JSON is structured and must support arbitrary JSON blobs, you can:
Wait for mypy to support recursive types.
If you can't wait, just use object or Dict[str, object]. It ends up being nearly identical to using recursive types in practice.
If you don't want to constantly have to type-check your code, use Any or Dict[str, Any]. Doing this lets you avoid needing to sprinkle in a bunch of isinstance checks or casts at the expense of type safety.
If you know precisely what your JSON data looks like, you can:
Use a TypedDict
Use a library like Pydantic to deserialize your JSON into an object
More discussion follows below.
Case 1: You do not know how your JSON is structured
Properly typing arbitrary JSON blobs is unfortunately awkward to do with PEP 484 types. This is partly because mypy (currently) lacks recursive types: this means that the best we can do is use types similar to the one you constructed.
(We can, however, make a few refinements to your type. In particular, json.Dumps(...) actually does not accept arbitrary iterables. A generator is a subtype of Iterable, for example, but json.dumps(...) will refuse to serialize generators. You probably want to use something like Sequence instead.)
That said, having access to recursive types may not end up helping that much either: in order to use such a type, you would need to start sprinkling in isinstance checks or casts into your code. For example:
JsonType = Union[None, int, str, bool, List[JsonType], Dict[JsonType]]
def load_config() -> JsonType:
# ...snip...
config = load_config()
assert isinstance(config, dict)
name = config["name"]
assert isinstance(name, str)
So if that's the case, do we really need the full precision of recursive types? In most cases, we can just use object or Dict[str, object] instead: the code we write at runtime is going to be nearly the same in either case.
For example, if we changed the example above to use JsonType = object, we would still end up needing both asserts.
Alternatively, if you find sprinkling in assert/isinstance checks to be unnecessary for your use case, a third option is to use Any or Dict[str, Any] and have your JSON be dynamically typed.
It's obviously less precise than the options presented above, but asking mypy to not type check uses of your JSON dict and relying on runtime exceptions instead can sometimes be more ergonomic in practice.
Case 2: You know how your JSON data will be structured
If you do not need to support arbitrary JSON blobs and can assume it forms a particular shape, we have a few more options.
The first option is to use TypedDicts instead. Basically, you construct a type explicitly specifying what a particular JSON blob is expected to look like and use that instead. This is more work to do, but can let you gain more type-safety.
The main disadvantage of using TypedDicts is that it's basically the equivalent of a giant cast in the end. For example, if you do:
from typing import TypedDict
import json
class Config(TypedDict):
name: str
env: str
with open("my-config.txt") as f:
config: Config = json.load(f)
...how do we know that my-config.txt actually matches this TypedDict?
Well, we don't, not for certain.
This can be fine if you have full control over where the JSON is coming from. In this case, it might be fine to not bother validating the incoming data: just having mypy check uses of your dict is good enough.
But if having runtime validation is important to you, your options are to either implement that validation logic yourself or use a 3rd party library that can do it on your behalf, such as Pydantic:
from pydantic import BaseModel
import json
class Config(BaseModel):
name: str
env: str
with open("my-config.txt") as f:
# The constructor will raise an exception at runtime
# if the input data does not match the schema
config = Config(**json.load(f))
The main advantage of using these types of libraries is that you get full type safety. You can also use object attribute syntax instead of dict lookups (e.g. do config.name instead of config["name"]), which is arguably more ergonomic.
The main disadvantage is doing this validation does add some runtime cost, since you're now scanning over the entire JSON blob. This might end up introducing some non-trivial slowdowns to your code if your JSON happens to contain a large quantity of data.
Converting your data into an object can also sometimes be a bit inconvenient, especially if you plan on converting it back into a dict later on.
There has been a lengthy discussion (https://github.com/python/typing/issues/182) about the possibility of introducing a JSONType; however, no definitive conclusion has yet been reached.
The current suggestion is to just define JSONType = t.Union[str, int, float, bool, None, t.Dict[str, t.Any], t.List[t.Any]] or something similar in your own code.
I am trying to setup a send to email on click with app maker, one of my objects that sends is the date. I have this code for pulling that data:
var todayDate = widget.parent.parent.descendants.Field.value
I then get this error when testing the date data entries:
Failed due to illegal value in property: 4
at (unknown)
at (unknown)
at Inbound_Call.Container.Form1.Form1Header.Form1Spinner.visible:-1:132
at (unknown)
at (unknown)
at (unknown)
at Inbound_Call.Container.Form1.Form1Footer.Form1SubmitButton.onClick:7:40
Assuming that you are trying to make server call to send an email, smth like this:
google.script.run.myServerFunction(myDate, otherParam1, otherParam2);
I can suggest that call fails due to App Script parameters constraints:
Legal parameters are JavaScript primitives like a Number, Boolean, String, or null, as well as JavaScript objects and arrays that are composed of primitives, objects, and arrays. A form element within the page is also legal as a parameter, but it must be the function’s only parameter. Requests fail if you attempt to pass a Date, Function, DOM element besides a form, or other prohibited type, including prohibited types inside objects or arrays. Objects that create circular references will also fail, and undefined fields within arrays become null....
I need to send a HTTP request (and get XML response) from Flash that looks similar to following:
http://example.com/somepath?data=1&data=2&data=3
I.e. having several parameters that share same name, but have different values.
Until now I used following code to make HTTP requests:
var resp:XML = new XML();
resp.onLoad = function(success:Boolean) {/*...*/};
resp.ignoreWhite = true;
var req:LoadVars = new LoadVars();
req["someParam1"] = 3;
req["someParam2"] = 12;
req.sendAndLoad("http://example.com/somepath", resp, "GET");
In this case this will not do: there will be only one parameter having last value.
What are my options? I'm using actionscript 2.
Added
I guess, I can do something like that:
var url:String = myCustomFunctionForBuildingRequestString();
var resp:XML = new XML();
resp.onLoad = function(success:Boolean) {/*...*/};
resp.load(url);
But in that case I am loosing ability to do POST requests. Any alternatives?
Changing request is not appropriate.
The standard http way of sending array data is
http://example.com/?data[0]=1&data[1]=2
But this isn't wrong either (added from comment):
http://example.com/?data[]=1&data[]=2
Sending more parameters with the same name like you're doing, in practice means that all but the last item should be ignored. This is because when reading variables, the server overwrites (in memory) any item that has the same name as that one, because renaming a variable isn't good practice and never was.
I don't know much AS (none :p) but you'd access it as a list or array or whatever data structures it has.
Although POST may be having multiple values for the same key, I'd be cautious using it, since some servers can't even properly handle that, which is probably why this isn't supported ... if you convert "duplicate" parameters to a list, the whole thing might start to choke, if a parameter comes in only once, and suddendly you wind up having a string or something ... but i guess you know what you're doing ...
I am sorry to say so, but what you want to do, is not possible in pure AS2 ... the only 2 classes available for HTTP are LoadVars and XML ... technically there's also loadVariables, but it will simply copy properties from the passed object into the request, which doesn't change your problem, since properties are unique ...
if you want to stick to AS2, you need an intermediary tier:
a server to forward your calls. if you have access to the server, then you create a new endpoint for AS2 clients, which will decode the requests and pass them to the normal endpoint.
use javascript. with flash.external::ExternalInterface you can call JavaScript code. You need to define a callback for when the operation is done, as well as a JavaScript function that you can call (there are other ways but this should suffice). Build the request string inside flash, pump it to JavaScript and let JavaScript send it to the server in a POST request and get the response back to flash through the callback.
up to you to decide which one is more work ...
side note: in AS3, you'd use flash.net::URLLoader with dataFormat set to flash.net::URLLoaderDataFormat.TEXT, and then again encode parameters to a string, and send them.
Disclaimer; I've never used Actionscript and have no means for testing this.
Putting the same variable name with several values on the query string is the standard way of sending multi-value variables (for example form checkboxes) to web servers. If LoadVars is capable of sending multiple values then it seems plausible that the values should be stored in an array:
req["someParam1"] = ["foo","bar","bas"];
There also seems to be a decode function to LoadVars, what happens if you try to import the query string you want into the object?:
req.decode("someParam1=foo&someParam1=bar&someParam1=bas");
You cannot use loadvars like this - because data can be either 1 or 2 or 3, not all of them at the same time.
You can either pass it as a comma separated list:
var req:LoadVars = new LoadVars();
req["data"] = "1,2,3";
or as an xml string, and parse it at the server. I am not familiar with manipulating xml in AS2, but this is how you'd do it in AS3:
var xml:XML = <root/>;
xml.appendChild(<data>1</data>);
xml.appendChild(<data>2</data>);
xml.appendChild(<data>3</data>);
//now pass it to loadvars
req["data"] = xml.toXMLString();
The string you send is:
<root>
<data>1</data>
<data>2</data>
<data>3</data>
</root>