I am streaming a Avro encoded file over the network from a S3 compliant object store and trying to read it and put it in some data-structure.
Issue: The issue I am facing sometimes ( one or two times in one / two days in test node when running continuously) is that half way through the file it hits this exception Invalid sync! in the nextRawBlock() method in DataFileStream class.
I would like to detect the root-cause of this and fix. I have been trying to
reproduce this in a test app but unable to do so successfully. I am looking for ideas on
what might potentially cause this ?
any better ways of reproducing this.
More details
a) The Avro file is not downloaded to disk , I get a handle to the file stream using S3ObjectInputStream and feed it to DataFileStream constructor
and then read from the stream directly.
b) The app tries to read records from the Avro encoded file in batches
of 500 records at a time.
c) The file contains a header section containing a Long count and a KV Map of String to Integer. After that it contains a array of records where each record contains a String and a long array. The schema uses Avro's union construct for enabling this.
d) Number of records in the file on average is around 5M
e) This entire download happens in separate thread and not in any user request.
f) The file is uploaded to the store by a separate process.
Other observation:
a) Upon failure the app closes the stream and tries to again download and read the stream. What I observe is this takes the node to a high oldgen state slowing down user requests.
Related
PFB Following questions:
1) In my local, I don't have a networkMap so the maxTransactionSize and maxMessageSize needs to be made part of the extraConfig in deployNodes for each node?
2) Let's say I have an Excel of 100MB which I Zip and then upload to Node using rpc.uploadAttachment the SecureHash received will now be added to a Tx. After Successful Completion of the TX will both parties have the attachment? or The Receiver will get the file only when he opens the attachment?
3) If it's when the receiver opens the attachment, it's requested from the sender, the file travels over the network and reaches to the receiver and is stored in the H2 DB for future use. If attachment is required later, the blob can directly be provided from the DB?
4) Now where, how does attachmentContentCacheSizeMegaBytes come into picture? Since we are already storing it in the H2 DB where is it used? as a blob limit to the node_attachment table?
5) Also, is the file ever stored in the file system ? at the time of upload to the node? does it get stored directly to the H2 DB?
The maxTransactionSize and maxMessageSize is set by the network operator, and individual nodes cannot modify it. This is for compatibility reasons. All the nodes on the network need to be able to handle the largest-possible transaction to ensure they can resolve any transactions they receive
The receiver node downloads the attachment immediately, and not when it first opens the attachment
N/A
The attachmentContentCacheSizeMegaBytes node configuration option is optional and specifies how much memory should be used to cache attachment contents in memory. It defaults to 10MB
The attachment is stored in the node's database as a blob when it is first uploaded
I am currently testing Tridion 2011 and am having problems creating multimedia components with uploaded content (as opposed to external).
I fill out the title, schema, multimedia type, select a file from my system then click save. I get a Saving item... information message then approximately 30 seconds later I will receive a The wait operation timed out message.
There doesn't appear to be any error messages in the C:\Program Files (x86)\Tridion\log directory. Looking at the event viewer I see the following information relating to the save action
Unable to save Component (tcm:4-738361).
The wait operation timed out
Error Code:
0x8004033F (-2147220673)
Call stack:
System.Data.SqlClient.SqlConnection.OnError(SqlException,Boolean,Action`1)
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException,Boolean,Action`1)
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject,Boolean,Boolean)
System.Data.SqlClient.TdsParser.TryRun(RunBehavior,SqlCommand,SqlDataReader,BulkCopySimpleResultSet,TdsParserStateObject,Boolean&)
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader,RunBehavior,String)
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior,RunBehavior,Boolean,Boolean,Int32,Task&,Boolean)
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior,RunBehavior,Boolean,String,TaskCompletionSource`1,Int32,Task&,Boolean)
System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1,String,Boolean,Int32,Boolean)
System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
Tridion.ContentManager.Data.AdoNet.Sql.SqlDatabaseUtilities.SetBinaryContent(Int32,Stream)
Tridion.ContentManager.Data.AdoNet.ContentManagement.ItemDataMapper.Tridion.ContentManager.Data.ContentManagement.IItemDataMapper.SetBinaryContent(Stream,TcmUri)
Tridion.ContentManager.ContentManagement.RepositoryLocalObject.SetBinaryContent(BinaryContent)
Tridion.ContentManager.ContentManagement.Component.OnSaved(SaveEventArgs)
Tridion.ContentManager.IdentifiableObject.Save(SaveEventArgs)
Tridion.ContentManager.ContentManagement.VersionedItem.Save(Boolean)
Tridion.ContentManager.ContentManagement.VersionedItem.Save()
Tridion.ContentManager.BLFacade.ContentManagement.VersionedItemFacade.UpdateAndCheckIn(UserContext,String,Boolean,Boolean)
XMLState.Save
Component.Save
I already have my timeout settings in the Content Manager Snap-In set to high values (more than 10 minutes) due to another issue.
The BINARIES table in the Content Manage Database is 25GB if that helps.
Any ideas? Thanks.
Edit 1
Following suggestions from Bart Koopman, my DBA has rebuilt the indexes but does not reckon the Transaction log has any impact on performance. The problem persists.
Edit 2
I have just found more details of the error
Unable to save Component (tcm:0-0-0).
Timeout expired.
The timeout period elapsed prior to completion of the operation or the server is not responding.
A database error occurred while executing Stored Procedure "EDA_ITEMS_UPDATEBINARYCONTENT".EDA_ITEMS_UPDATEBINARYCONTENT
After taking a look at this procedure it looks like the following statement could be the root cause
SELECT 1 FROM BINARIES WHERE ID = #iBINARY_ID AND CONTENT IS NULL
I execute it manually with #iBINARY_ID as -1 and after 2 minutes it still hasn't completed. I assume that when I insert a new multimedia component the query will be something similar (i.e. the id will not exist in the table).
The BINARIES table currently has a NON-CLUSTERED Primary Key. Maybe the solution would be to change this to a CLUSTERED Primary Key? However, I assume it is NON-CLUSTERED for a reason.
Just had a response from SDL customer support. Apparently this is a known issue related to statistics and the chosen query plan.
Running the following statement manually from SQL Server Management Studio fixes the problem (it didn't even need to complete for me)
SELECT 1 FROM BINARIES WHERE ID = -1 AND CONTENT IS NULL
Hope this helps someone else out!
Timeouts on database operations are usually an indication of a misconfiguration or a lack of maintenance. By increasing the timeout you are just working around the problem rather than solving it.
With a binaries table that big you will want to make sure you have proper database setup with data files that are separated from your log files (separated on different physical partitions/disks) and possibly even multiple data files on multiple physical partitions to take advantage of performance gains.
Next to that you will want to assure that the standard database maintenance is performed daily/hourly. Things like backing up and truncating the transaction log every hour will greatly improve your database performance (on MS SQL Server a transaction log of more than 1GB slows the database down drastically, you should always try to keep it below that size through timely backup/trucate). Updating statistics and rebuilding indexes is also something you should not forget on a regular basis.
I'm using Core Data API in my iOS application. Also, I'm using the commoncrypto library (CCCrypt()) to encrypt/decrypt the database file(.sqlite file) that resides in the documents folder when the application state changes (Background/Foreground).
The problem which i'm facing is... some of the records in the database gets lost when the application is killed manually by the user from the background state and this issue is inconsistent.
I'm just converting the sqlite file contents into NSData and used as an input to CCCrypt() function to encrypt/decrypt and I'm not decoding the any input data in the crypt operation.
Can someone please help me .....what could be reason for the data loss ? that too only when the application gets killed manually from the background state ..... For both encryption and decryption operation, the CCCrypt function returns the status as kCCSuccess...
Does it needed to use any sort of decoding the input data (raw bytes) before CCCrypt operation ?
Probably you should use the life cycle methods to save your data before it gets killed.
Try saving data in applicationWillTerminate. Go through this discussion for more details about
[enter link description here][1]
[1]: applicationWillTerminate when is it called and when not"Save data before getting Killed"
I am using sqlite in my application only for read access. The DB gets hit often by my application and I could see that the header(100 bytes) of the database is read every time when i access the database.
Precisely speaking, 16 bytes from the 24th byte of the header is read everytime. My question is , if the database is used only for read purpose, why the header is read everytime as the database connection is not closed?..can we make it read it only once?
Thanks!!
Google search gave me this link, and it says
"Your process may promise that it will only read the database, but there
might be some other process writing to it.
Not being a server, sqlite has no other way to find that out than by
reading the header over and over again. It has to check whether the
schema was changed, or whatever other info is in those bytes."
http://www.mail-archive.com/sqlite-users#sqlite.org/msg69900.html
1.
I have created a audio player in silverlight.
within that player user is able to select a portion of song to save as ringtone.
but i got the time duration from .. but I have to cut the partial portion of stream or audio stream and save it to the server dick.
Plz suggest me how I can convert the selected audio time duration into the stream or byte array..?
-- Additional information on this question:
2.
I have created a ringtone audio player in silverlight. Within that user can select a portion which can be cut and save as a audio file.
I am unable to save the stream to the disk.. it is giving following errors:
Error 1. Attempt to access the method failed System IO FileInfo OpenWrite
Plz help
-- Additional information on this question:
3.
What are the use of MediaStreamSample & MediaStreamSource class in silverlight with respect to MediaElement?
Will it help in cutting a portion of audio file in order to create the ringtone out of a song?
If you need to save to the server, then you need to get that data to the server.
Just saving it (as answered) will try to save to the client's machine. What you need to do is upload the data to the server either via a WCF service or an ASHX handler or such. I've done something similar -- uploading MP3 files from a Silverlight client to a WCF service via a Stream, works well.
Next: You need to make sure that whatever splitting process you use accommodates the audio format-- ie you probably just can't split the binary file. What format are you using, mp3?
I've used something called mp3plt, before to split mp3s. You may be able to recompile the source into a Silverlight-compatible library, assuming it's written in something you can use, source here.
Or you can look into the mp3 specs to see if it is possible to just split the binary file, in which case taking the duration to cut (the one the user chose), and multiplying by the bitrate, (kb/s * seconds = kb) will give you the place in the file byte[] you can cut at.
Error 1. Attempt to access the method failed System IO FileInfo OpenWrit, you getting this error coz of security reasons. Before saving to disk you should promt SaveFileDialog to user, and then only save file to disk.