Nexus: Unknown Task called "Storage facet cleanup" - nexus

I installed Sonatype Nexus OSS 3.6.2-01
and although I did not configured any Scheduled Tasks I can see in the commandline that Nexus is running a task called "Storage facet cleanup" every ten minutes.
Because of I could not find anything about this task I finally came up here.
2017-12-20 08:40:00,013+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change WAITING -> RUNNING
2017-12-20 08:40:00,051+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change RUNNING -> WAITING (OK)
2017-12-20 08:50:00,014+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change WAITING -> RUNNING
2017-12-20 08:50:00,029+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change RUNNING -> WAITING (OK)
2017-12-20 09:00:00,015+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change WAITING -> RUNNING
2017-12-20 09:00:00,031+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change RUNNING -> WAITING (OK)
2017-12-20 09:10:00,016+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change WAITING -> RUNNING
2017-12-20 09:10:00,032+0100 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change RUNNING -> WAITING (OK)
What is this task for?
Why is it not visible in the "Tasks" Section of the admin-panel?
Is there a way to disable this task (or configure it running only at night)?

This is a cleanup task which is required to have NXRM run correctly. It is not listed because it's not disablable/configurable.

This is a cleanup task defined in the class StorageFacetCleanupTask which run performDeletions() on the storageFacetManager:
#Override
protected Void execute() throws Exception {
thread = Thread.currentThread();
long count;
do {
count = storageFacetManager.performDeletions();
}
while (count > 0 && !isCanceled());
return null;
}
The method performDeletions is defined in StorageFacetManagerImpl and it delete all buckets marked with P_PENDING_DELETION:
#Override
#Guarded(by = STARTED)
public long performDeletions() {
List<Bucket> buckets = findBucketsForDeletion();
return buckets.stream().filter((bucket) -> {
try {
log.info("Deleting bucket for repository {}", bucket.getRepositoryName());
deleteBucket(bucket);
return true;
}
catch (Exception e) {
log.warn("Unable to delete bucket with repository name {}, will require manual cleanup",
bucket.getRepositoryName(), e);
return false;
}
}).count();
}
All buckets marked with P_PENDING_DELETION are collected by findBucketsForDeletion in StorageFacetManagerImpl:
private List<Bucket> findBucketsForDeletion() {
return inTx(databaseInstanceProvider).call(db -> {
return StreamSupport
.stream(bucketEntityAdapter.browse(db).spliterator(), false)
.filter((bucket) -> bucket.attributes().contains(P_PENDING_DELETION))
.collect(Collectors.toList());
});
}

Related

Spring Kafka got an infinite retry attempt

Spring Kafka got an infinite retry attempt
Version: Spring-Kafka-2.2.4.RELEASE
Listener setting
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setRetryTemplate(getRetryTemplate());
factory.setConcurrency(concurrency);
return factory;
}
And using a RetryTemplate with SimpleRetryPolicy for setting the MaxAttempts
Kafka Listener
#KafkaListener(topics = "topic")
public void myListener(ConsumerRecord<String, String> record) {
//Code here
}
Some strange issue is happening on server with multiple instance running. So whenever myListener throwing an exception, the message will be retried up to the max attempt and ACK'ed fine on local.
But on the server, the following is happen:
Node A: Consume #1 -> Retry #1 -> Retry #1 -> Consume #2 -> Retry #2 -> Retry #2 -> Consume #3 ....
Node B:...................................... Consume #1 -> Retry #1 -> Retry #1 -> Consume #2 ....
And so on without any ACK. After 500 Message (ex: Consume #500), node A reset again back to Consume #1. Not sure if the number #500 is having any relation to max.poll.records
Still can't reproducing the issue on local setup since it's working fine (ACK after maxAttempt).
But is there any kind of explanation/solution for this problem? currently the shortcut is just catch the exception and prevent the retrying mechanism so the message can be ACK and offset moving forwards.
Thanks!

Spring for Apache Kafka idleBetweenPolls param leads to continuous rebalance

I've got Spring for Apache Kafka KafkaListenerContainerFactory bean configured with idleBetweenPolls param:
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, Event>> dlqKafkaListenerContainerFactory(
Map<String, Object> kafkaConfigs
) {
kafkaConfigs.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 10 * 60 * 1000);
var factory = new ConcurrentKafkaListenerContainerFactory<String, Event2Extended<?>>();
factory.setConsumerFactory(new DefaultKafkaConsumerFactory<>(kafkaConfigs));
factory.getContainerProperties().setIdleBetweenPolls(5 * 60 * 1000L);
factory.setErrorHandler(new SeekToCurrentErrorHandler(new FixedBackOff(0L, 0L)));
factory.setMessageConverter(new StringJsonMessageConverter());
return factory;
}
My application is running in 4 pods in OpenShift and I have a continuous rebalancing problem.
Problem logs by o.a.k.c.c.internals.AbstractCoordinatorare:
Attempt to heartbeat with Generation...and group instance id Optional.empty failed due to UNKNOWN_MEMBER_ID, resetting generation
Attempt to heartbeat failed since group is rebalancing
(Re-)joining group
But the problem disappears if I remove:
factory.getContainerProperties().setIdleBetweenPolls(5 * 60 * 1000L);

.Net-Core 2.2 MassTransit.ConfigurationException: The state machine was not properly configured

Newbie question - what am I missing? Are there any dotnetcore 2.2 Saga examples available?
I have a basic end to end system working OK with messages flowing across containers in docker-compose, but adding a Saga seems to be a challenge -
Q. Am I missing a scheduler dependency? In MassTransit 5.5.5, cfg.UseInMemoryMessageScheduler(); doesn't compile.
Something odd was going on, I had to mark my state machine explicitly as ISaga
MassTransit.ConfigurationException: Failed to create the state machine connector for Model.WorkflowExecutionStateMachine ---> MassTransit.ConfigurationException: The state machine was not properly configured:
workflowapi_1 | [Failure] ExecutingTask was not specified
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
// Register MassTransit
services.AddMassTransit(x =>
{
x.AddConsumer<WorkflowTaskConsumer>();
// required?
x.AddSaga<WorkflowExecutionSaga>();
x.AddBus(provider => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var rabbitMQHostName = $"rabbitmq://{configuration["RabbitMQHostName"]}";
Console.Out.WriteLineAsync($"Starting Workflow Receiver... {rabbitMQHostName}/{QueueNames.ExeuteWorkflowTaskQueue}");
var host = cfg.Host(new Uri(rabbitMQHostName), hostConfig =>
{
hostConfig.Username("guest");
hostConfig.Password("guest");
});
// A basic message works OK
cfg.ReceiveEndpoint(host, QueueNames.ExeuteWorkflowTaskQueue, ep =>
{
ep.PrefetchCount = 1;
ep.UseMessageRetry(mr => mr.Interval(1000, 2));
ep.ConfigureConsumer<WorkflowTaskConsumer>(provider);
});
// Doesn't like this
cfg.ReceiveEndpoint(host, QueueNames.WorkflowStateMachineSagaQueueName, ep =>
{
ep.PrefetchCount = 1;
ep.UseMessageRetry(mr => mr.Interval(1000, 2));
ep.StateMachineSaga(new WorkflowExecutionSaga(), new InMemorySagaRepository<WorkflowExecutionStateMachine>());
});
}));
cfg.UseInMemoryMessageScheduler(); // doesn't compile!
});
}
Bus is started as follows -
public void Configure(IApplicationBuilder app, IHostingEnvironment env, IApplicationLifetime lifetime)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
// The default HSTS value is 30 days. You may want to change this for production scenarios,
// see https://aka.ms/aspnetcore-hsts.
app.UseHsts();
}
app.UseMvc();
var bus = app.ApplicationServices.GetService<IBusControl>();
var busHandle = TaskUtil.Await(() =>
{
return bus.StartAsync();
});
lifetime.ApplicationStopping.Register(() =>
{
busHandle.Stop();
});
}
Exception details are
Unhandled Exception: MassTransit.ConfigurationException: Failed to create the state machine connector for Rapid.Workflow.Api.Model.WorkflowExecutionStateMachine ---> MassTransit.ConfigurationException: The state machine was not properly configured:
workflowapi_1 | [Failure] ExecutingTask was not specified
workflowapi_1 | at Automatonymous.StateMachineConfigurationResult.CompileResults(IEnumerable1 results)
workflowapi_1 | at Automatonymous.StateMachineConnectors.StateMachineConnector1.StateMachineEvents()+MoveNext()
workflowapi_1 | at System.Collections.Generic.List1.AddEnumerable(IEnumerable1 enumerable)
workflowapi_1 | at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
workflowapi_1 | at Automatonymous.StateMachineConnectors.StateMachineConnector1..ctor(SagaStateMachine1 stateMachine)
workflowapi_1 | --- End of inner exception stack trace ---
workflowapi_1 | at Automatonymous.StateMachineConnectors.StateMachineConnector1..ctor(SagaStateMachine1 stateMachine)
workflowapi_1 | at Automatonymous.SagaConfigurators.StateMachineSagaConfigurator1..ctor(SagaStateMachine1 stateMachine, ISagaRepository1 repository, ISagaConfigurationObserver observer)
workflowapi_1 | at MassTransit.AutomatonymousReceiveEndpointExtensions.StateMachineSaga[TInstance](IReceiveEndpointConfigurator configurator, SagaStateMachine1 stateMachine, ISagaRepository1 repository, Action`1 configure)
workflowapi_1 | at Rapid.Workflow.Api.Startup.<>c.b__2_5(IRabbitMqReceiveEndpointConfigurator ep) in /src/Workflow.Api/Startup.cs:line 74
Dependencies are
<PackageReference Include="Automatonymous" Version="4.1.6" />
<PackageReference Include="MassTransit" Version="5.5.5" />
<PackageReference Include="MassTransit.RabbitMQ" Version="5.5.5" />
<PackageReference Include="MassTransit.AspNetCore" Version="5.5.5" />
<PackageReference Include="MassTransit.Automatonymous" Version="5.5.5" />
<PackageReference Include="MassTransit.Extensions.DependencyInjection" Version="5.5.5" />
<PackageReference Include="Newtonsoft.Json" Version="12.0.2" />
Thanks for any tips or ideas -
You need to change use the .AddStateMachineSaga method, instead of the .AddSaga method you're using in the code in question.
// required? - yes, but should be as shown here
x.AddSagaStateMachine<WorkflowExecutionSaga, WorkflowState>();
In this case, both the state machine and the state machine instance type are required. Then, in your endpoint, use:
ep.ConfigureSaga<WorkflowState>(provider);
You also need to make sure you have the saga repository configured in the container, which is done for MS DI/in-memory using:
x.AddSingleton<ISagaRepository<WorkflowState>, InMemorySagaRepository<WorkflowState>>();
That should get you rolling, assuming your state machine isn't broken. If you still get the error, make sure all your state machine events, etc. are properly configured.
Also, your state machine instance should implement:
public class WorkflowState :
SagaStateMachineInstance
And your state machine does not need to implement ISaga.
public class WorkflowExecutionSaga :
MassTransitStateMachine<WorkflowState>
This error seems to crop up because the Saga class had declared some as-yet-unused (but public) events - DOH!
The solution was to remove unused events from the Saga...
// uncomment will fail! public Event<ISatelliteTaskRequest> UnusedEvent { get; private set; }
After looking at this sample https://github.com/selcukusta/masstransit-saga-implementation and cutting my program.cs back to basics - I was still getting the error! So, not a container / IOC / Startup issue.
Next, looking in source for the MassTransit error message (https://github.com/MassTransit/MassTransit/blob/master/src/MassTransit.AutomatonymousIntegration/Configuration/StateMachineConnectors/StateMachineConnector.cs) I realised that the correlating code is possibly reflecting on all public members of the Saga -
So, removing unused events from the Saga class fixes the issue.

Broken connection with MariaDB

I have a Wildly server connected to MariaDB Server on the same machine.
For some reason I get this error from time to time:
20:38:51,536 INFO [stdout] (default task-1) 20:38:51.535 [default task-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08
20:38:51,536 INFO [stdout] (default task-1) 20:38:51.536 [default task-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - (conn=25) Connection reset by peer (Write failed)
20:38:51,549 INFO [stdout] (default task-1) 20:38:51.548 [default task-1] ERROR o.s.t.i.TransactionInterceptor - Application exception overridden by rollback exception
20:38:51,550 INFO [stdout] (default task-1) org.springframework.dao.DataAccessResourceFailureException: could not prepare statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not prepare statement
20:38:51,550 INFO [stdout] (default task-1) at deployment.datalis_admin.war//org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:275)
20:38:51,550 INFO [stdout] (default task-1) at deployment.datalis_admin.war//org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:253)
20:38:51,550 INFO [stdout] (default task-1) at deployment.datalis_admin.war//org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:527)
In MariaDB log I get: 2019-03-06 20:27:51 25 [Warning] Aborted connection 25 to db: 'production_gateway' user: 'wildfly' host: 'localhost' (Got timeout reading communication packets)
Do you know what might be the issue and how to fix it?
POM file:
https://pastebin.com/HUNy0ULy
application.properties:
spring.datasource.jndi-name=java:/global/production_gateway
spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MariaDBDialect
spring.jpa.show-sql = true
spring.jpa.hibernate.ddl-auto = update
JPA configuration:
import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;
#Configuration
#EnableTransactionManagement
public class ContextDatasource {
#Bean
public EntityManager entityManager(EntityManagerFactory emf) {
return emf.createEntityManager();
}
#Bean
public PlatformTransactionManager transactionManager(final EntityManagerFactory emf) {
final JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(emf);
return transactionManager;
}
#Bean
public PersistenceExceptionTranslationPostProcessor exceptionTranslation() {
return new PersistenceExceptionTranslationPostProcessor();
}
}
JDBC driver:
mariadb-java-client-2.4.0.jar
MariaDB version:
mysql Ver 15.1 Distrib 10.3.13-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
Can you advice?
Please show us your pom.xml and watch out if your driver's version is compatible for your DB version as well. Some they have to update the driver web the DB developer make some change.

CompletableFuture.exceptionally with executor

CompletableFuture.exceptionally() method takes a lambda, but there is no flavor of the method that takes a custom Executor, or even an "...Async" flavor of it.
Which executor does exceptionally lambda run on? Would it be the same executor which ran the original CompletableFuture which threw the exception? Or (I would be surprised if this is the case) is it the commonPool ?
Form JDK bug discussion CompletableFuture.exceptionally may execute on main thread :
CompletableFuture.exceptionally does not take an Executor argument
since it is not designed to execute the exceptionally task
asynchronously.
If the dependent task is not yet completed then the exceptionally task
will complete on the same thread that dependent tasks completes on.
If the dependent task is completed then the exceptionally task will
complete on the thread that executed the call to exceptionally.
This is the same behaviour that will occur for any non-asynchronous
execution, such as thenAccept.
To guarantee that the exceptionally task is executed on a thread
within the executor thread pool then it is necessary to use
whenCompleteAsync or handleAsync and passing in the executor.
Note that as of JDK 12, there is CompletionStage.exceptionallyAsync (and exceptionallyAsync which takes an Executor).
Looks like it runs in the same executor as its CompletionStage
public static void main(String[] args) throws Exception {
//ExecutorService executorService = Executors.newFixedThreadPool(3);
ExecutorService executorService = Executors.newWorkStealingPool();
while (true) {
int i = Instant.now().getNano();
CompletableFuture<?> completableFuture = CompletableFuture.supplyAsync(
()-> {
System.err.printf("async thread at %d -> %s\n", i, Thread.currentThread().getName());
try {
TimeUnit.SECONDS.sleep(2);
} catch (InterruptedException e) {
e.printStackTrace();
}
throw new RuntimeException();
}, executorService);
completableFuture.exceptionally(
(err)-> {
System.err.printf("error thread for %d -> %s\n", i, Thread.currentThread().getName());
return null;
});
TimeUnit.SECONDS.sleep(5);
}
}
For fixed size:
async thread at 418000000 -> pool-1-thread-1
error thread for 418000000 -> pool-1-thread-1
async thread at 646000000 -> pool-1-thread-2
error thread for 646000000 -> pool-1-thread-2
async thread at 646000000 -> pool-1-thread-3
error thread for 646000000 -> pool-1-thread-3
async thread at 646000000 -> pool-1-thread-1
error thread for 646000000 -> pool-1-thread-1
async thread at 647000000 -> pool-1-thread-2
error thread for 647000000 -> pool-1-thread-2
For stealing pool(4 cores):
async thread at 96000000 -> ForkJoinPool-1-worker-1
error thread for 96000000 -> ForkJoinPool-1-worker-1
async thread at 196000000 -> ForkJoinPool-1-worker-1
error thread for 196000000 -> ForkJoinPool-1-worker-1
async thread at 197000000 -> ForkJoinPool-1-worker-1
error thread for 197000000 -> ForkJoinPool-1-worker-1
async thread at 197000000 -> ForkJoinPool-1-worker-1
error thread for 197000000 -> ForkJoinPool-1-worker-1
async thread at 197000000 -> ForkJoinPool-1-worker-1
error thread for 197000000 -> ForkJoinPool-1-worker-1
async thread at 197000000 -> ForkJoinPool-1-worker-1
with no executor:
async thread at 848000000 -> ForkJoinPool.commonPool-worker-1
error thread for 848000000 -> ForkJoinPool.commonPool-worker-1
async thread at 944000000 -> ForkJoinPool.commonPool-worker-1
error thread for 944000000 -> ForkJoinPool.commonPool-worker-1
async thread at 944000000 -> ForkJoinPool.commonPool-worker-1
error thread for 944000000 -> ForkJoinPool.commonPool-worker-1
async thread at 944000000 -> ForkJoinPool.commonPool-worker-1
error thread for 944000000 -> ForkJoinPool.commonPool-worker-1
async thread at 944000000 -> ForkJoinPool.commonPool-worker-1
error thread for 944000000 -> ForkJoinPool.commonPool-worker-1
I did some experiments. Looks like the executor for the exceptionally handler is not chosen deterministically.
If I do not place any breakpoints, the exceptionally handler is occasionally executed in the same executor as the CompletableFuture that has the exceptionally handler. Furthermore, I could notice that it ran in the same ForkJoinTask. Once in two or three runs, it was executed on the main thread.
If I placed breakpoint at runtime prior to attaching the exceptionally handler and waited there for a moment, the exceptionally lambda was consistently executed on the main (calling) thread!
Here is the experiment code:
ForkJoinPool ex = new ForkJoinPool(2,
ForkJoinPool.defaultForkJoinWorkerThreadFactory, null, false);
// AsyncThreadLocal.requestId.set((int)(Math.random()*1000));
CompletableFuture<Integer> f1 = CompletableFuture.supplyAsync(() -> {
System.out.println(Thread.currentThread().getName());
throw new RuntimeException();
// return 3;
}, ex);
CompletableFuture<Integer> f8 = f1.exceptionally(t -> {
System.out.println(Thread.currentThread().getName());
return 5;
});
Thread.sleep(10000);
Output of program: Sometimes it is
ForkJoinPool-1-worker-1
main
Other times it is
ForkJoinPool-1-worker-1
ForkJoinPool-1-worker-1

Resources