Connect to non-standard couchbase ports for integration testing - integration-testing

I'm trying to set up automated integration-tests for a Spring Boot application that connects to couchbase. I want to test against a fresh containerised database using random available ports so that we don't need to worry about port collisions on developer machines or in continuous integration environments.
So far I've created a test that uses com.palantir.docker.compose.DockerComposeRule to spin up a couchbase docker image that exposes the standard ports on random available ports. However, org.springframework.boot.autoconfigure.couchbase.CouchbaseProperties doesn't expose any properties for ports.
I can see that org.springframework.data.couchbase.config.CouchbaseEnvironmentParser suggests that there is some way to set the ports. I haven't been able to work out how to use it though. Has anyone else been able to do this?
This is what my test looks like so far:
#Slf4j
#RunWith(SpringRunner.class)
#SpringBootTest
#Category(IntegrationTest.class)
public class BiscuitRepositoryIT {
private static final int COUCHBASE_PORT = 8091;
private static final String COUCHBASE_SERVICE = "couchbase";
#ClassRule
public static DockerComposeRule docker = DockerComposeRule.builder()
.file("src/test/resources/docker-compose-couchbase.yml")
.projectName(ProjectName.random())
.waitingForService(COUCHBASE_SERVICE, HealthChecks.toHaveAllPortsOpen())
.build();
#Autowired
private BiscuitRepositoryConfig config;
#Autowired
private BiscuitRepository biscuitRepository;
#BeforeClass
public static void initialize() {
DockerPort couchbase = docker.containers()
.container(COUCHBASE_SERVICE)
.port(COUCHBASE_PORT);
log.info("couchbase ports: {}", couchbase);
// TODO update integration-test properties file with the ports
// before the spring context starts.
}
#After
public void tearDown() throws Exception {
biscuitRepository.deleteAll();
}
#Test
public void insertBiscuit() throws Exception {
Biscuit digestive = Biscuit.builder().name("digestive").manufacturer("biscuitCorp").build();
Biscuit persistedBiscuit = biscuitRepository.save(digestive);
assertThat(persistedBiscuit).isEqualToIgnoringGivenFields(digestive, "id", "version");
}
}

Related

spring-boot embedded kafka issue : I am getting Invalid receive (size = 369296129 larger than 104857600)

I am writing integration test cases using spring boot, embedded kafka, and temporal. I am trying to send a message on a kafka topic.
#SpringBootTest(classes = Application.class)
#RunWith(SpringJUnit4ClassRunner.class)
#ActiveProfiles("test")
#DirtiesContext
#EmbeddedKafka(
partitions = 5,
controlledShutdown = true,
brokerProperties = {
"listeners=PLAINTEXT://localhost:9092",
"port=9092"
})
public class OutboundFlowIT {
private final Logger logger = LoggerFactory.getLogger(OutboundFlowIT.class);
private TestWorkflowEnvironment testEnv;
private Worker worker;
private WorkflowClient workflowClient;
#Autowired
private ActivityService activityService;
#Autowired
private EventSender sender;
#Before
public void setUp(){
// some setup code.
}
#Test
public void processOutboundFinancialMessage_shouldTriggerAllSteps_WhenOK() throws IOException,InterruptedException {
// logic for sending message to intended topic.
}
But I am getting below error.
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369296129 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105) ~[kafka-clients-2.5.1.jar:na]
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:447) ~[kafka-clients-2.5.1.jar:na]
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:397) ~[kafka-clients-2.5.1.jar:na]
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678) ~[kafka-clients-2.5.1.jar:na]
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580) ~[kafka-clients-2.5.1.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:485) ~[kafka-clients-2.5.1.jar:na]
at kafka.network.Processor.poll(SocketServer.scala:861) ~[kafka_2.12-2.5.1.jar:na]
at kafka.network.Processor.run(SocketServer.scala:760) ~[kafka_2.12-2.5.1.jar:na]
I have also added the below configurations in kafka.properties but I am getting the same issue as above.
spring.kafka.producer.properties.max.request.size=569296129
spring.kafka.consumer.properties.max.partition.fetch.bytes=369296129
I am new to kafka , please help me.
How do you send the messages to the kafka broker? You should use the kafka protocol and not using HTTP request, resemble issues states this is the error comes from

Calling seek of ConsumConsumerSeekCallback from a Spring Boot application

Here is my setup:
ConsumerSeekAware implementation:
public class ReplayJobKafkaConsumer implements ConsumerSeekAware, AcknowledgingMessageListener<String, String> {
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) {
}
#Override
public void onIdleContainer(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) {
}
private static final ThreadLocal<ConsumerSeekCallback> seekCallBack = new ThreadLocal<>();
private static ConsumerSeekCallback consumerSeekCallback;;
#Override
public void registerSeekCallback(ConsumerSeekCallback callback) {
this.seekCallBack.set(callback);
consumerSeekCallback = callback;
}
public void onMessage(final ConsumerRecord<String, String> data, final Acknowledgment acknowledgment) {
}
public static ThreadLocal<ConsumerSeekCallback> getSeekCallback(){
return seekCallBack;
}
public static ConsumerSeekCallback getAnotherSeekCallback(){
return consumerSeekCallback;
}
}
My Spring Boot application approximates to:
#SpringBootApplication
public class ReplayJobApplication{
...
public void run(final String... args){
context = SpringApplication.run(ReplayJobApplication.class, args);
ReplayJobKafkaConsumer.getAnotherSeekCallback().seek("top", 0, 23);
}
...}
The above setup works. Now I can run this application using
java -jar -Dstart.offset=0....
But it only works if the seekcallback variable is not a ThreadLocal. I need this to be accessible at the Spring Boot application as that is how I intend running this consumer. TEMP-TOPIC's other consumers can still be processing, but I intend to run this consumer on a need basis with a start and end offset. While the command line parameters can be read in the consumer, the concerns I have are
callback variable is static (I cannot possibly create an instance of ReplayJobKafkaConsumer
it is a plain variable and not a ThreadLocal
Though the life time of this container is only going to be from start to end, I wonder if this setup is flawed and need some confirmation that this implementation is OK.
You appear to have some fundamental misunderstanding of what's going on.
The ThreadLocal is needed because the Kafka consumer object is not thread-safe. If you store the callback in a ThreadLocal, you can perform arbitrary seek operations at runtime - either from the onMessage method, or by listening for an ListenerContainerIdleEvent when there are no messages.
You can't perform arbitrary seeks ReplayJobKafkaConsumer.getAnotherSeekCallback().seek("top", 0, 23); from another thread.
You can't perform arbitrary seeks before partitions have been assigned.
So, as I have been telling you in other answers/comments, you must do the seek when the partition(s) are assigned.
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> map, ConsumerSeekCallback consumerSeekCallback) {
// Do the seeks here using the `consumerSeekCallback` parameter.
}
With modern versions of spring-kafka, you don't need to use ConsumerSeekAware unless you want to perform arbitrary seeks at runtime (after the initial seek). You can use a ConsumerAwareRebalanceListener instead.

Handle on db not released during unit test global setup

I'm using NUnit 3 to do a global setup, which creates a local database needed to run several of my service tests, which looks like this:
[SetUpFixture]
public class FixtureSetup
{
private MobileServiceClient _client;
private SyncService _syncService;
[OneTimeSetUp]
public void GlobalSetup()
{
_client = Substitute.For<MobileServiceClient>(Settings.SyncUrl);
_syncService = Substitute.For<SyncService>(_client);
}
[OneTimeTearDown]
public void GlobalTeardown()
{
_syncService = null;
_client.Dispose();
}
}
Settings.SyncUrl contains the URL to Azure to which the Azure App Services SDK will by syncing eventually, and is not relevant to this question.
The one-time setup, simply constructs a new instance of the MobileServiceClient and passes that instance to my SyncService class, to construct the local store, which looks like this:
public class SyncService : ISyncService
{
private readonly IMobileServiceClient _client;
private MobileServiceSQLiteStore Store { get; }
public SyncService(IMobileServiceClient client)
{
_client = client;
Store = new MobileServiceSQLiteStore(Settings.SyncDb);
Store.DefineTable<User>();
_client.SyncContext.InitializeAsync(Store);
}
public async Task<List<TTable>> All<TTable>()
{
var table = await _client.GetSyncTable<TTable>().ToListAsync();
return table;
}
public async Task<TTable> Insert<TTable>(TTable table)
{
await _client.GetSyncTable<TTable>().InsertAsync(table);
return table;
}
public async Task<List<TTable>> Search<TTable>(Expression<Func<TTable, bool>> predicate)
{
var table = await _client.GetSyncTable<TTable>().Where(predicate).ToListAsync();
return table;
}
}
Settings.SyncDb simply points to the name of the db, called localstorage.db, and if on a mobile device, will store this in the application's file repository, on Windows or Mac, it will store it under the user's profile folder. Adding this just for reference.
My problem is that the global setup creates the localstorage.db correctly, but by the time the unit test runs, it cannot access the localstorage.db, because it's seemingly still in use by the global setup method.
I thought that reinstantiating the MobileServiceClient in the test class would resolve this, but it does not seem to do so. Is there a way that I can release the handle on the db, before hitting the unit test?
This is not an issue in development, as I can run the unit tests again after the first fail, but VSTS builds fail the test due to this reason.
Thanks in advance.

FeignClients get published as REST endpoints in spring cloud application

I've got REST FeignClient defined in my application:
#FeignClient(name = "gateway", configuration = FeignAuthConfig.class)
public interface AccountsClient extends Accounts {
}
I share endpoint interface between server and client:
#RequestMapping(API_PATH)
public interface Accounts {
#PostMapping(path = "/register",
produces = APPLICATION_JSON_VALUE,
consumes = APPLICATION_JSON_VALUE)
ResponseEntity<?> registerAccount(#RequestBody ManagedPassUserVM managedUserDTO)
throws EmailAlreadyInUseException, UsernameAlreadyInUseException, URISyntaxException;
}
Everythng works fine except that my FeignClient definition in my client application also got registered as independent REST endpoint.
At the moment I try to prevent this behavior using filter which returns 404 status code for FeignClinet client mappings in my client application. However this workeraund seems very inelegant.
Is there another way how to prevent feign clients registering as separate REST endpoints?
It is a known limitation of Spring Cloud's feign support. By adding #RequestMapping to the interface, Spring MVC (not Spring Cloud) assumes you want as an endpoint. #RequestMapping on Feign interfaces is not currently supported.
I've used workaround for this faulty Spring Framework behavior:
#Configuration
#ConditionalOnClass({Feign.class})
public class FeignMappingDefaultConfiguration {
#Bean
public WebMvcRegistrations feignWebRegistrations() {
return new WebMvcRegistrationsAdapter() {
#Override
public RequestMappingHandlerMapping getRequestMappingHandlerMapping() {
return new FeignFilterRequestMappingHandlerMapping();
}
};
}
private static class FeignFilterRequestMappingHandlerMapping extends RequestMappingHandlerMapping {
#Override
protected boolean isHandler(Class<?> beanType) {
return super.isHandler(beanType) && (AnnotationUtils.findAnnotation(beanType, FeignClient.class) == null);
}
}
}
I found it in SpringCloud issue

Implementing WebSocket on Spring MVC based Server

I searched a lot about implementing WebSocket/XMPP on Spring MVC based server but couldn't reach to a concrete answer. So here is my requirement
Receive a message from a client (in my case it will be a android/iOS mobile) via WebSocket/XMPP on tomcat server and parse the actual message at server side
Send a message from server app to WebSocket/XMPP client
If somebody could help me to point on some good tutorial or demo code, it would be a great help.
run Tomee 1.5.2
http://openejb.apache.org/downloads.html
activate the ActiveMQ JMS Server. create an OpenEJB configuration.
http://www.mail-archive.com/users#openejb.apache.org/msg04327.html
setup an XMPP ActiveMQ server protocol listener (in the activemq.xml)
in your Spring services configuration, create a Spring JMS listener (Spring ListenerContainer) configuration on the Topic/Queue.
you can use the JmsTemplate to push a message out to the Queue/Topic via ActiveMQ, the XMPP client will receive the message.
Enjoy!
BTW: This is exactly what I am in the middle of setting up right now...still learning.
check this out: www.xchat.io. It was built based on Asynchronous Spring MVC (DefferredResult, you know), XMPP, and jQuery. it's promising.
I am not sure if this is just perfect way to achieve or not, but for now I have found a solution and it would be glad to share it here.
There are two steps that you have to done.
1. Instead of ordinary HTTPServlet sub class, create a sub class of WebSocketServlet and
2. Create a sub class of MessageInbound class and override its required methods.
P.S. : Only latest version of tomcat supports WebSocket (apache tomcat 7.0.42 or higher).
Here is a WebSocket class.
public class WsChatServlet extends WebSocketServlet {
private static final long serialVersionUID = 1456546233L;
#Override
protected StreamInbound createWebSocketInbound(String protocol,
HttpServletRequest request) {
return new IncomingMessageHandler();
}
}
And this is a simple class which can send/receive message (String/binary).
public class IncomingMessageHandler extends MessageInbound {
#Override
public void onOpen(WsOutbound outbound) {
logger.info("Open Client.");
}
#Override
public void onClose(int status) {
logger.info("Close Client.");
}
#Override
public void onTextMessage(CharBuffer cb) throws IOException {
logger.info("Text Message received:" + cb.toString());
}
#Override
public void onBinaryMessage(ByteBuffer bb) throws IOException {
}
public synchronized void sendTextMessage(String message) {
try {
CharBuffer buffer = CharBuffer.wrap(message);
this.getMyoutbound().writeTextMessage(buffer);
this.getMyoutbound().flush();
} catch (IOException e) {
// Handle Exception
}
}
}

Resources