Given an object like this:
Matcher matcher = pattern.matcher(sql);
with usage like so:
Set<String> matches = new HashSet<>();
while (matcher.find()) {
matches.add(matcher.group());
}
I'd like to replace this while loop by something more object-oriented like so:
new Iterator<String>() {
#Override
public boolean hasNext() {
return matcher.find();
}
#Override
public String next() {
return matcher.group();
}
}
so that I can easily e.g. make a Stream of matches, stick to using fluent APIs and such.
The thing is, I don't know and can't find a more concise way to create this Stream or Iterator. An anonymous class like above is too verbose for my taste.
I had hoped to find something like IteratorFactory.from(matcher::find, matcher::group) or StreamSupport.of(matcher::find, matcher::group) in the jdk, but so far no luck. I've no doubt libraries like apache commons or guava provide something for this, but let's say I can't use those.
Is there a convenient factory for Streams or Iterators that takes a hasNext/next method combo in the jdk?
In java-9 you could do it via:
Set<String> result = matcher.results()
.map(MatchResult::group)
.collect(Collectors.toSet());
System.out.println(result);
In java-8 you would need a back-port for this, taken from Holger's fabulous answer
EDIT
There is a single method btw tryAdvance that could incorporate find/group, something like this:
static class MyIterator extends AbstractSpliterator<String> {
private Matcher matcher;
public MyIterator(Matcher matcher) {
// I can't think of a better way to estimate the size here
// may be you can figure a better one here
super(matcher.regionEnd() - matcher.regionStart(), 0);
this.matcher = matcher;
}
#Override
public boolean tryAdvance(Consumer<? super String> action) {
while (matcher.find()) {
action.accept(matcher.group());
return true;
}
return false;
}
}
And usage for example:
Pattern p = Pattern.compile("\\d");
Matcher m = p.matcher("12345");
Set<String> result = StreamSupport.stream(new MyIterator(m), false)
.collect(Collectors.toSet());
This class I wrote embodies what I wanted to find in the jdk. Apparently though it just doesn't exist. eugene's accepted answer offers a java 9 Stream solution though.
public static class SearchingIterator<T> implements Iterator<T> {
private final BooleanSupplier advancer;
private final Supplier<T> getter;
private Optional<T> next;
public SearchingIterator(BooleanSupplier advancer, Supplier<T> getter) {
this.advancer = advancer;
this.getter = getter;
search();
}
private void search() {
boolean hasNext = advancer.getAsBoolean();
next = hasNext ? Optional.of(getter.get()) : Optional.empty();
}
#Override
public boolean hasNext() {
return next.isPresent();
}
#Override
public T next() {
T current = next.orElseThrow(IllegalStateException::new);
search();
return current;
}
}
Usage:
Matcher matcher = Pattern.compile("\\d").matcher("123");
Iterator<String> it = new SearchingIterator<>(matcher::find, matcher::group);
Related
There is a quite simple case I would like to implement:
I have a base and DLT topics:
MessageBus:
Topic: my_topic
DltTopic: my_dlt_topic
Broker: event-serv:9092
So, those topics are already predefined, I don't need to create them automatically.
The only I need to handle broken messages automatically without retries, because they don't make any sense, so I have something like this:
#KafkaListener(topics = ["#{config.messageBus.topic}"], groupId = "group_id")
#RetryableTopic(
dltStrategy = DltStrategy.FAIL_ON_ERROR,
autoCreateTopics = "false",
attempts = "1"
)
#Throws(IOException::class)
fun consume(rawMessage: String?) {
...
}
#DltHandler
fun processMessage(rawMessage: String?) {
kafkaTemplate.send(config.messageBus.dltTopic, rawMessage)
}
That of course doesn't work properly.
I also tried to specify a kafkaTemplate
#Bean
fun kafkaTemplate(
config: Config,
producerFactory: ProducerFactory<String, String>
): KafkaTemplate<String, String> {
val template = KafkaTemplate(producerFactory)
template.defaultTopic = config.messageBus.dltTopic
return template
}
however, that does not change the situation.
In the end, I believe there is an obvious solution, so I please give me a hint about it.
See the documenation.
#SpringBootApplication
public class So69317126Application {
public static void main(String[] args) {
SpringApplication.run(So69317126Application.class, args);
}
#RetryableTopic(attempts = "1", autoCreateTopics = "false", dltStrategy = DltStrategy.FAIL_ON_ERROR)
#KafkaListener(id = "so69317126", topics = "so69317126")
void listen(String in) {
System.out.println(in);
throw new RuntimeException();
}
#DltHandler
void handler(String in) {
System.out.println("DLT: " + in);
}
#Bean
RetryTopicNamesProviderFactory namer() {
return new RetryTopicNamesProviderFactory() {
#Override
public RetryTopicNamesProvider createRetryTopicNamesProvider(Properties properties) {
if (properties.isMainEndpoint()) {
return new SuffixingRetryTopicNamesProviderFactory.SuffixingRetryTopicNamesProvider(properties) {
#Override
public String getTopicName(String topic) {
return "so69317126";
}
};
}
else if(properties.isDltTopic()) {
return new SuffixingRetryTopicNamesProviderFactory.SuffixingRetryTopicNamesProvider(properties) {
#Override
public String getTopicName(String topic) {
return "so69317126.DLT";
}
};
}
else {
throw new IllegalStateException("Shouldn't get here - attempts is only 1");
}
}
};
}
}
so69317126: partitions assigned: [so69317126-0]
so69317126-dlt: partitions assigned: [so69317126.DLT-0]
foo
DLT: foo
This is a Kafka server configuration so you must set it on the server. The relevant property is:
auto.create.topics.enable (true by default)
I have a problem with mocking cause it keep calling the original function. This is my demo code
First file is interface that contains the function that I want to mock.
public interface IDemoReplace
{
int FunctionToBeReplaced();
}
Second file is a class that actually has the implementation for the function
public class DemoReplace : IDemoReplace
{
public int FunctionToBeReplaced()
{
//this function contains sql query in my real project
return 1;
}
}
Third file is a class that I want to test
public class ClassToBeTested
{
public int TestThisFunction()
{
IDemoReplace replace = new DemoReplace();
var temp = replace.FunctionToBeReplaced();
return temp;
}
}
Last file is the test class
public class TestClass
{
[Fact]
public void TryTest()
{
using (var mock = AutoMock.GetLoose()) {
//Arrange
mock.Mock<IDemoReplace>()
.Setup(x => x.FunctionToBeReplaced())
.Returns(returnTwo());
var classToBeTested = mock.Create<ClassToBeTested>();
var expected = 2;
//Act
var actual = classToBeTested.TestThisFunction();
//Assert
Assert.Equal(expected, actual);
}
}
public int returnTwo() {
return 2;
}
}
This test will be failed with expected is 2 and actual is 1. When I tried to debug it doesn't call returnTwo but call the original function instead.
I am new to unit testing so what did I miss? Please be considered that the code above is only a demo of what is happened in my actual project. FunctionToBeReplaced is actually a function that execute and return record from database so I want to mock that function.
Thanks :)
This is a design issue. The subject under test is tight coupled to implementation concerns that make it difficult to isolation the subject so that it can be unit tested.
It (subject) is manually creating its dependency
IDemoReplace replace = new DemoReplace();
Ideally you want to explicitly inject dependencies. Those dependencies should also be abstractions and not concretions.
public class ClassToBeTested {
private readonly IDemoReplace dependency;
public ClassToBeTested(IDemoReplace dependency) {
this.dependency = dependency;
}
public int TestThisFunction() { ;
var temp = dependency.FunctionToBeReplaced();
return temp;
}
}
At run time, the implementation (or mock) can be injected, either purely, or via a container.
The test in the original example shown should now behave as expected.
public class TestClass {
[Fact]
public void TryTest() {
using (var mock = AutoMock.GetLoose()) {
//Arrange
var expected = returnTwo();
mock.Mock<IDemoReplace>()
.Setup(x => x.FunctionToBeReplaced())
.Returns(expected);
var classToBeTested = mock.Create<ClassToBeTested>();
//Act
var actual = classToBeTested.TestThisFunction();
//Assert
Assert.Equal(expected, actual);
}
}
public int returnTwo() {
return 2;
}
}
We are using Storm to process streaming data and store into HDFS. We have got everything to work but have one issue. I understand that we can specify the number of tuples after which the data gets flushed to HDFS using SyncPolicy, something like this below:
SyncPolicy syncPolicy = new CountSyncPolicy(Integer.parseInt(args[3]));
The question I have is can the data also be flushed after a timeout? For e.g. we have set the SyncPolicy above to 1000 tuples. If for whatever reason we get 995 tuples and then the data stops coming in for a while is there any way that storm can flush the 995 records to HDFS after a specified timeout (5 seconds)?
Thanks in advance for any help on this!
Shay
Yes, if you send a tick tuple to the HDFS bolt, it will cause the bolt to try to sync to the HDFS file system. All this happens in the HDFS bolt's execute function.
To configure tick tuples for your topology, in your topology config. In Java, to set that to every 300 seconds the code would look like:
Config topologyConfig = new Config();
topologyConfig.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 300);
StormSubmitter.submitTopology("mytopology", topologyConfig, builder.createTopology());
You'll have to adjust that last line depending on your circumstances.
There is an alternative solution for this problem,
First, lets clarify about sync policy, If your sync policy is 1000 ,then HdfsBolt only sync the data from 1000 tuple by calling hsync() method in execute() means it only clears the buffer by pushing data to disk, but for faster write disk may uses its cache and not writing to file directly.
The data is written to the file only when the size of data matches your rotation policy that need to specify at the time of bolt creation.
FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(100.0f, Units.KB);
So for flushing the record the to file after timeout, Seperate your tick tuple from normal tuples in excecute method and calculate the time difference of both tuple, If the diff is greater than timeout period then write the data to file.
By handling tick tuple differently you can also avoid the tick tuple frequency written to your file.
See the below code for better understanding:
public class CustomHdfsBolt1 extends AbstractHdfsBolt {
private static final Logger LOG = LoggerFactory.getLogger(CustomHdfsBolt1.class);
private transient FSDataOutputStream out;
private RecordFormat format;
private long offset = 0L;
private int tickTupleCount = 0;
private String type;
private long normalTupleTime;
private long tickTupleTime;
public CustomHdfsBolt1() {
}
public CustomHdfsBolt1(String type) {
this.type = type;
}
public CustomHdfsBolt1 withFsUrl(String fsUrl) {
this.fsUrl = fsUrl;
return this;
}
public CustomHdfsBolt1 withConfigKey(String configKey) {
this.configKey = configKey;
return this;
}
public CustomHdfsBolt1 withFileNameFormat(FileNameFormat fileNameFormat) {
this.fileNameFormat = fileNameFormat;
return this;
}
public CustomHdfsBolt1 withRecordFormat(RecordFormat format) {
this.format = format;
return this;
}
public CustomHdfsBolt1 withSyncPolicy(SyncPolicy syncPolicy) {
this.syncPolicy = syncPolicy;
return this;
}
public CustomHdfsBolt1 withRotationPolicy(FileRotationPolicy rotationPolicy) {
this.rotationPolicy = rotationPolicy;
return this;
}
public CustomHdfsBolt1 addRotationAction(RotationAction action) {
this.rotationActions.add(action);
return this;
}
protected static boolean isTickTuple(Tuple tuple) {
return tuple.getSourceComponent().equals(Constants.SYSTEM_COMPONENT_ID)
&& tuple.getSourceStreamId().equals(Constants.SYSTEM_TICK_STREAM_ID);
}
public void execute(Tuple tuple) {
try {
if (isTickTuple(tuple)) {
tickTupleTime = Calendar.getInstance().getTimeInMillis();
long timeDiff = normalTupleTime - tickTupleTime;
long diffInSeconds = TimeUnit.MILLISECONDS.toSeconds(timeDiff);
if (diffInSeconds > 5) { // specify the value you want.
this.rotateWithOutFileSize(tuple);
}
} else {
normalTupleTime = Calendar.getInstance().getTimeInMillis();
this.rotateWithFileSize(tuple);
}
} catch (IOException var6) {
LOG.warn("write/sync failed.", var6);
this.collector.fail(tuple);
}
}
public void rotateWithFileSize(Tuple tuple) throws IOException {
syncHdfs(tuple);
this.collector.ack(tuple);
if (this.rotationPolicy.mark(tuple, this.offset)) {
this.rotateOutputFile();
this.offset = 0L;
this.rotationPolicy.reset();
}
}
public void rotateWithOutFileSize(Tuple tuple) throws IOException {
syncHdfs(tuple);
this.collector.ack(tuple);
this.rotateOutputFile();
this.offset = 0L;
this.rotationPolicy.reset();
}
public void syncHdfs(Tuple tuple) throws IOException {
byte[] e = this.format.format(tuple);
synchronized (this.writeLock) {
this.out.write(e);
this.offset += (long) e.length;
if (this.syncPolicy.mark(tuple, this.offset)) {
if (this.out instanceof HdfsDataOutputStream) {
((HdfsDataOutputStream) this.out).hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH));
} else {
this.out.hsync();
}
this.syncPolicy.reset();
}
}
}
public void closeOutputFile() throws IOException {
this.out.close();
}
public void doPrepare(Map conf, TopologyContext topologyContext, OutputCollector collector) throws IOException {
LOG.info("Preparing HDFS Bolt...");
this.fs = FileSystem.get(URI.create(this.fsUrl), this.hdfsConfig);
this.tickTupleCount = 0;
this.normalTupleTime = 0;
this.tickTupleTime = 0;
}
public Path createOutputFile() throws IOException {
Path path = new Path(this.fileNameFormat.getPath(),
this.fileNameFormat.getName((long) this.rotation, System.currentTimeMillis()));
this.out = this.fs.create(path);
return path;
}
}
You can directly use this class in your project.
Thanks,
I am writing a Metro App.
I am trying to read a file and return a float[] from the data. But no matter what I do, the function seems to return null. I have tried the solutions to similar questions to no luck.
For example if I use:
float[] floatArray = new ModelReader("filename.txt").ReadModel()
The result will be a null array.
However if I use:
new ModelReader("filename.txt")
The correct array will be printed to the console because "Test" also prints the array before returning it. This seems very weird to me.
Please give me some guidance, I have no idea what is wrong.
public class ModelReader
{
float[] array;
public ModelReader(String name)
{
ReadModelAsync(name);
}
public float[] ReadModel()
{
return array;
}
private async Task ReadModelAsync(String name)
{
await readFile(name);
}
async Task readFile(String name)
{
// settings
var path = #"Assets\models\" + name;
var folder = Windows.ApplicationModel.Package.Current.InstalledLocation;
// acquire file
var file = await folder.GetFileAsync(path);
// read content
var read = await Windows.Storage.FileIO.ReadTextAsync(file);
using (StringReader sr = new StringReader(read))
{
Test test = new Test(getFloatArray(sr));
this.array = test.printArray();
}
}
private float[] getFloatArray(StringReader sr) { ... }
public class Test
{
public float[] floatArray;
public Test(float[] floatArray)
{
this.floatArray = floatArray;
}
public float[] printArray()
{
for (int i = 0; i < floatArray.Length; i++)
{
Debug.WriteLine(floatArray[i]);
}
return floatArray;
}
}
You're trying to get the result of an asynchronous operation before it has completed. I recommend you read my intro to async / await and follow-up with the async / await FAQ.
In particular, your constructor:
public ModelReader(String name)
{
ReadModelAsync(name);
}
is returning before ReadModelAsync is complete. Since constructors cannot be asynchronous, I recommend you use an asynchronous factory or asynchronous lazy initialization as described on my blog (also available in my AsyncEx library).
Here's a simple example using an asynchronous factory approach:
public class ModelReader
{
float[] array;
private ModelReader()
{
}
public static async Task<ModelReader> Create(string name)
{
var ret = new ModelReader();
await ret.ReadModelAsync(name);
return ret;
}
...
}
I've been trying to implement form validation correctly and a discussion on fubu mailing list has been the most helpful (http://groups.google.com/group/fubumvc-devel/browse_thread/thread/d54b135fe0254653/12180cd86e9dc50b).
I'm still not entirely clear on certain points, I'm a newbie so I'm going through some yak shaving.
It seems like the example given in the discussion performed the validation within the controller itself using IsValid(model).
I'm trying to avoid this by decorating my input model with validation attributes such as Required and then use the validation configuration to Transfer on failure (via a policy).
this.Validation(x => {
x.Actions
.Include(call => call.HasInput && call.InputType().Name.EndsWith("Input"));
x.Failures
.ApplyPolicy<AccountValidationFailedPolicy>();
});
And here's the class that implments the policy:
public class AccountValidationFailedPolicy : IValidationFailurePolicy {
public bool Matches(ValidationFailure context) {
return (context.InputType() == typeof (RegisterAccountInput));
}
public void Handle(ValidationFailure context) {
var incomingRequest = (RegisterAccountInput) context.InputModel;
var failedValidation = new RegisterationFailedNotification {
CVV = incomingRequest.CVV,
AcceptTerms = incomingRequest.AcceptTerms,
Countries = incomingRequest.Countries,
PhoneNumber = incomingRequest.PhoneNumber,
PIN = incomingRequest.PIN
};
FubuContinuation.TransferTo(failedValidation);
}
}
Handle simply tries to Transfer to another action via a new model, copying the values into the new model so that I can redisplay them again on the form.
I must be doing something wrong here, because it's not transferring anywhere.
I have a class with this method which I was hoping would handle it.
public AccountViewModel New(RegisterationFailedNotification notification) {
....
}
Am I on track here, or is there something fundamental that I'm not getting? Perhaps a policy is not the thing to do here?
#stantona
The policy mechanism will work here. I'll spare you the details about how I plan to make this simpler (very soon), and note that your use of FubuContinuation.TransferTo simply creates a FubuContinuation -- it doesn't execute it.
Here's what you need:
public class AccountValidationFailedPolicy : IValidationFailurePolicy {
private readonly IFubuRequest _request;
private readonly IValidationContinuationHandler _handler;
public AccountValidationFailedPolicy(IFubuRequest request, IValidationContinuationHandler handler) {
_request = request;
_handler = handler;
}
public bool Matches(ValidationFailure context) {
return (context.InputType() == typeof (RegisterAccountInput));
}
public void Handle(ValidationFailure context) {
var incomingRequest = (RegisterAccountInput) context.InputModel;
var failedValidation = new RegisterationFailedNotification {
CVV = incomingRequest.CVV,
AcceptTerms = incomingRequest.AcceptTerms,
Countries = incomingRequest.Countries,
PhoneNumber = incomingRequest.PhoneNumber,
PIN = incomingRequest.PIN
};
var continuation = FubuContinuation.TransferTo(failedValidation);
_request.Set(continuation);
_handler.Handle();
}
}