I am using solr-5.4.0 in my production environment(solr cloud mode) and am trying to automate the reload/restart process of the solr collections based on certain specific conditions.
I noticed that on solr reload the thread count increases a lot there by resulting in increased latencies. So I read about reload process and came to know that while reloading
1) Solr creates a new core internally and then assigns this core same name as the old core. Is this correct?
2) If above is true then does solr actually create a new index internally on reload?
3) If so then restart sounds much better than reload, or is there any better way to upload new configs on solr?
4) Can you point me to any docs that can give me more details about this?
Any help would be appreciated, Thank you.
If I'm not wrong you wanted to restart/reload the collection in production (Solr cloud mode) and asking for the best approach, If that's true here are few points for the consideration-
If possible, Could you please provide more details like what it cause/requirement to reload/restart the collection in production
I’m assuming the reason could be to refresh the shared resource (Like to see the changes of updated synonyms, or adding or deleting a stop word, etc.) or to update the Solr config set.
Here are a few points for the consideration-
If you want to update the shared resources –
Upload the resources through Solr API and reload the collection through (https://lucene.apache.org/solr/guide/6_6/collections-api.html#CollectionsAPI-Input.1)
If you want to update the config set –
When running in SolrCloud mode, changes made to the schema on one node will propagate to all replicas in the collection. You can pass the updateTimeoutSecs parameter with your request to set the number of seconds to wait until all models confirm they applied the schema updates. ( I got this information from solr-5.4.0, and it’s similar to what we have in Solr 6.6 here https://lucene.apache.org/solr/guide/6_6/schema-api.html)
1) Solr creates a new core internally and then assigns this same core name as the old core. Is this correct?
Not sure about it. Can you please share some reference?
2) If above is true then does solr create a new index internally on reload?
Not sure about it. Can you please share some reference?
3) If so then restart sounds much better than reload, or is there any better way to upload new configs on solr?
I don’t agree, because ideally reload is part of the restart, as per my understanding there will be an additional process in reset related to caching and sync.
4) Can you point me to any docs that can give me more details about this?
Here is a link for reference guide- https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-5.4.pdf
Related
I have a .NET 6 application that writes logs out to a SQLite file each time a cycle is completed. I'm using EF Core.
The application sits on a Raspberry Pi with limited resource, so I want to keep the live database small as when it gets large the system slows down. So I want to archive the logs to only keep the last 50 or so logs in the live database.
I am hoping to create a method that will will create a new SQLite database file and then copy the last oldest record over when a new log is created. I'd also want to limit how big the archive file is, maybe split out to create a new one once it reaches a certain size on disk.
I've had a look around and can't find any best practices anything documented. Could someone give me a steer to the best way to achieve this or similar.
I solved my issue by putting EFCore aside and instead using the sqlite-net-pcl nuget package from SQLite-net.
This allowed me to separate the creation of my archive and also apply additional commands not supported in EFCore like vacuum.
I still use EFCore and Linq to query the records to create my archive with and then again to remove those records once the archive is created.
This question is like " What was first, chicken or egg?".
Let's imagine we have some source code. Written using symfony or yii. It has db migration code that hadle some database changes.
Now, we have some commits that updates our code (for example new classes) and some db changes (change old columns or add new tables).
When we developing at localhost or update our dev servers it's ok to have time to stop services\any actions and update server. But when we tries to do it on production server we will crash everything for a while and this is not an option.
Why this will happen - when we pull it (git\mercurial) our code will be updated, but NOT database, and when code will be executed - it will throw exceptions of database. To fix it we should run build-in framework migrations. So in the end our server will be crashed until migrations will be called.
Code and migrations should be updated "in one time".
What is the best practice to handle it?
ADDED:
Solution like "run pull then run migrations in one call" - not an option in highload project. Because on highload even in second some entries\calls can be borken.
Stop server we cannot too.
Pulling off a zero downtime deployment can be a bit tricky and there are many ways to achieve this.
As for the database it is recommended to do changes in a backwards compatible fashion. So for example adding a nullable column or new table will not affect your existing code base and can be done safely. So if you want to add a new non-nullable column you would do it in 3 steps:
Add new column as nullable
Populate with data to make sure there are no null-values
Make the column NOT NULL
You will need a new deployment for 1 & 3 at the very least. When modifying a column it's pretty much the same, you create a new column, transfer the data over, release the code that uses the new column (optionally with the old column as fallback) and then remove the old column (plus fallback code) in a 3rd deployment.
This way you make sure that your database changes will not cause a downtime in your existing application. This takes great care and obviously requires you to have a good deployment pipeline allowing for fast releases. If it takes hours to get a release out this method will not be fun.
You could copy the database (or even the whole system), do a migration and then switch to that instance, but in most applications this is not feasible because it will make it a pain to keep both instances in sync between deployments. I cannot recommend investing too much time in that, but I might be biased from my experience.
When it comes to switching the current version of your code with a newer one you have multiple options. The fancy cloud based solutions like kubernetes make this kind of easy. You create a second cluster with your new version and then slowly route traffic from the old cluster to the new one. If you have a single server it is quite common to deploy a new release to a separate folder, do all the management tasks like warming caches and then when the release is ready to be used you switch a symlink to the newest release. Both methods require meticulous planning and tweaking if you really want them to be zero downtime. There are all kinds of thing that can cause issues like a shared cache being accidentally cleared to sessions not being transferred over correctly to the new release. Whenever something that's stored in a session changes you have to take a similar approach as with the database and basically slow move the state over to the new one while running the code or having a fallback to still handle the old data otherwise you might get errors when reading the session, causing 500 pages for your customers.
They key to deploy with as few outages and glitches as possible is good monitoring of the systems and the application to see where things go wrong during a deployment to make it more stable over time.
You can create a backup server with content that mirrors your current server. Then do some error detection.
If an error is detected on your primary server, update your DNS record to divert your traffic to your secondary server.
Once primary back up and running, traffic moves back to primary and then sync the changes in your secondary.
These are called failover servers.
I have found so far an bundle that uses memcache as translation source but I haven't found anything on how to move the translation cache from disk storage to a service or directly to memcache.
I have also look at the options for the framework but I haven'T found anything useful on it (or I'm to stupid to use google ^^).
I need to move the cache files to memcache for deployment reason.
I'm having multiple Application Servers.
And to store the translation cache etc. on disk is slow and pane full if I deploy software (php Process on the productive app servers need to be restarted). It would make my live easier if that stuff would be stored in memcache as I would simply flush memcache to reset the translation stuff.
did anyone ever try this?
What first comes to the mind is to make a console command that would use one Loader (for example, \Symfony\Component\Translation\Loader\XliffFileLoader) and then another Dumper (something implementing \Symfony\Component\Translation\Dumper\DumperInterface from that bundle, like MemcacheDumper).
In your command your would load translations from one source by loader (in the form of \Symfony\Component\Translation\MessageCatalogue) and then dump them into another.
I'm working with Azure's offline-sync API.
(It's REALLY GREAT so far, but since it's still new-ish it doesn't have comprehensive documentation, only tutorials. We need to craft dependable integration tests, and we're finding that tricky because we need to rely on published behavior in official docs for that... or dig into the source, but that is liable to change at any time.)
The samples do this:
var store = new MobileServiceSQLiteStore("localstore.db");
The comments mention "initializes local store".
I assume the local sync database is a "throw-away" asset, as it can be recreated at will.
Is the expected behavior that it will create the local SQLite file if it does not exist, or it will recreate the file each time the mobile app starts and that call is made?
The tutorials are augmented by the HOWTO documentation (available under Mobile > Develop - in the same area as the tutorials) and the GitHub Wiki and the github.io pages for the SDK.
The local store is created if it doesn't exist, and new fields are added to tables if they are needed. It's sometimes good to delete the database - for example, if you reduce the field count in your mobile app (the process only adds fields). If you do this, the database will be re-created when the app is next restarted.
I'm using alfresco throw cmis.
On one of our environment, we have an issue.
We want to create a folder and put some docs in it.
This works fines in all our env except one.
In this one, we can create the folder.
But when we do a search to find the folder, the folder isn't found.
After that i can find it with the share gui.
I have no error message in the share app.
Does any one have an idea on what could be the issue?
Promoting a comment to an answer...
When using Alfresco with SOLR, you need to be aware that the SOLR index isn't quite real-time. Close to real time, sure, but it's asynchronous so there's always a lag. (It's an eventually consistent index, not a fully realtime one)
There's a lot of information on the Alfresco and SOLR Wiki, including the way you can query what the current lag is.
If the lag is very low (eg a lightly loaded system), you can find that SOLR will catch up almost instantly, and newly created items will show instantly in the search results. However, it's more normal to expect to have to wait a little bit, especially on more loaded systems.
If no new results are showing up even after several minutes, you'll want to follow the instructions on the wiki or the SOLR Monitoring and Troubleshooting docs to work out why and fix.