I have a Symfony 5.4 app with an endpoint that fetches some data from db:
$response = $this->em->getConnection()->executeQuery($sql, $params)->fetchAllAssociative();
And then outputs to json (or to a csv with similar results):
return new JsonResponse($response);
The problem is that it goes out of memory when the data starts to increase, even if there isn't so much data in the db (the whole schema is 600MB with 2 years of data, and it reaches 500mb memory when querying for last month's data).
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes)
How to optimize this action to use less memory?
An example with 50k rows limit:
dump(memory_get_usage() / 1000000); // 9MB
$db = $this->em->getConnection();
dump(memory_get_usage() / 1000000); // 11MB
$response = $db->executeQuery($sql, $params)->fetchAllAssociative();
dump(memory_get_usage() / 1000000); // 64MB
$response = new JsonResponse($response);
dump(memory_get_usage() / 1000000); // 23MB
dump(strlen($response->getContent()) / 1000000); // 11MB (response body)
Related
I'm doing some synthetic testing of Kafka. My goal is to learn what maximum throughput I can achieve in my specific setup. The problem I'm getting is that after increasing throughput to a certain level (by gradually starting more producer containers) and sustaining that throughput level for 10-15 seconds individual producer container throughput slows downs (drops from 16K mps to 8-12K mps, consumed message rate reduces in line with the producer rate) and one of the two things happen:
either producers fail with Local: Queue full error
or Kafka container crashes.
While message rate is sustained, Kafka memory consumption slowly grows from 500Mb to 1.5Gb. Kafka instance CPU usage (as reported by docker stats command is around 60-70%, which I believe translates to 0.6-0.7 CPU in Docker configuration terms). Producer and consumer memory and CPU load is uneventful.
I've also noticed that reducing message size allows to sustain the message rate longer, but the next rate increase leads to the same symptoms.
My initial suspicions were related to amounts of memory available to Kafka (not sure how to validate this suspicion as Kafka is not logging any exception before crashing) so I have significantly reduced retention.ms from 300K to 5K in the hope that this would reduce amount of memory needed by Kafka to maintain the message rate, but this has not helped.
What could be causing the issue? Any steps to help debug the issue are also highly appreciated!
The setup looks like this:
Host machine:
CPU: 8 Cores, 16 threads
RAM: 64 Gb
OS: PopOS
1 Kafka container
2 CPU (docker-compose deploy > resources > limits > cpus setting)
2 Gb RAM (docker-compose deploy > resources > limits > memory, memswap_limit and Kafka -Xmx and -Xms settings)
1 Topic, 40 partitions, retention.ms = 5000
4 Producer containers (.Net Confluent.Kafka package)
0.5 CPU
2 Gb RAM
1 producer instance per container
10 threads per producer
Thread message rate: 1600 per second
Batch size: 128
LingerMs: 40
Combined produced message rate: 4 x 10 x 1600 = 64K mps
Message size: 1Kb
Using ProduceAsync API method
Considering message sent when async DeliveryResult task returned by ProduceAsync has been completed
2 Consumer containers (.Net Confluent.Kafka package)
1 CPU
2 Gb RAM
1 consumer instance per container
20 threads per consumer
all consumer belong to a single group (so each message is consumed once by one of the consumers and not by each of the consumers)
Core producer code used:
static void Produce(int id, ProducerConfig config, double sendInterval, Options o) {
Console.WriteLine($"Starting thread: {id}, sleepInterval: {sendInterval}, batchSize: {o.BatchSize}");
string msg = new string('A', o.MsgSize);
using (var producer = new ProducerBuilder<Null, string>(config).Build()) {
while (true) {
for (int i = 0; i < o.BatchSize; i++) {
var t = producer.ProduceAsync(o.Topic, new Message<Null, string> { Value = msg });
Interlocked.Increment(ref messagesSent);
t.ContinueWith(task => {
if (task.IsFaulted) {
Console.WriteLine($"{t.Exception}");
} else {
Interlocked.Increment(ref messagesDelivered);
}
});
}
Thread.Sleep((int)sendInterval);
}
}
}
Core consumer code used:
...
for (int i = 0; i < o.ThreadCount; i++) {
int id = RandomNumberGenerator.GetInt32(10000);
ConsumerConfig threadConfig = new ConsumerConfig(config);
threadConfig.GroupInstanceId = id.ToString();
var consumer = new ConsumerBuilder<Ignore, string>(threadConfig).Build();
consumers.Add(consumer);
Thread t = new Thread(() => { Consume(id, consumer, token, o); });
t.Start();
threads.Add(t);
}
...
static void Consume(int id, Confluent.Kafka.IConsumer<Ignore, string> consumer, CancellationToken token, Options o) {
Console.WriteLine($"Starting thread: {id}");
try {
consumer.Subscribe(o.Topic);
while (!token.IsCancellationRequested) {
var consumeResult = consumer.Consume(token);
Interlocked.Increment(ref messagesReceived);
}
} catch (OperationCanceledException) {
Console.WriteLine($"Thread cancelled: {id}");
} finally {
consumer.Close();
}
Console.WriteLine($"Ending thread: {id}");
}
Kafka service definition:
kafka:
image: 'bitnami/kafka:3.1.1'
container_name: kafka
ports:
- '9092:9092'
- '6666:6666'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
- KAFKA_CFG_LISTENERS=PLAINTEXT://:29092,PLAINTEXT_HOST://:9092
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
- KAFKA_HEAP_OPTS=-Xmx${MEM_MED_JAVA} -Xms${MEM_MED_JAVA}
- JMX_PORT=6666
- KAFKA_CFG_BROKER_ID=1
depends_on:
- zookeeper
deploy:
resources:
limits:
cpus: ${CPU_LARGE}
memory: ${MEM_MED}
memswap_limit: ${MEM_MED}
#GetMapping("/")
#ResponseBody
public void getInvoice(#RequestParam String DocumentId,
HttpServletResponse response) {
DocumentDAO documentDAO = null;
try {
documentDAO = service.downloadDocument(DocumentId);
response.setContentType("application/" + documentDAO.getSubtype());
IOUtils.copy(documentDAO.getDocument(), response.getOutputStream());
response.flushBuffer();
documentDAO.getDocument().close();
} catch (IOException e) {
e.printStackTrace();
}
}
The task is to stream pdf document from back-end server (a lot of big documents, up to 200 MB) to the browser via SpringMVC controller. Back-end server outputs document in InputStream - I am copying it to response OutputStream.
IOUtils.copy(documentDAO.getDocument(), response.getOutputStream());
And it works. I just do not like java memory consumption on machine, where this SpringMVC is running.
If it is streaming - why memory consumption increases very high, when customer performs request to this mvc controller?
When big document (i.e. 100 mb) is requested - java heap size increases accordingly.
What I expect is - my java machine should use only some buffer sized amount of memory should not load document to memory, just stream it by.
Is my expectation wrong? Or it is correct and I should do something somehow different?
Thank you in advance.
Here is graph of memory increase when requesting 63MB document
https://i.imgur.com/2fjiHVB.png
and then repeating the request after a while
https://i.imgur.com/Kc77nGM.png
and then GC does its job at the end
https://i.imgur.com/WeIvgoT.png
I have to sync file with firebase when it changes. I know how to set up handler when file changes and then call Firebase.set. But what will happen if file is quite big (9mb) and it takes 30 seconds for example for set operation to finish and in the mean time file changes again? What happens when set on some locations is in progress and second set on the same location is performed?
Fb will automatically cancel in-progress operation and start it again?
Or it will wait till in-progress opeartion is finished and then start it again?
Or it will be the best to monitor if set on goven location is if progress, and if it is then queue next set operation after the in-progress finishes?
sample test
ref1 = new Firebase('https://my_firebase.firebaseio.com/some_location')
ref2 = new Firebase('https://my_firebase.firebaseio.com/some_location')
ref1.set({ some_large_data: 'abcef...' });
ref2.set({ some_large_data: '12345...' });
// which set will take effect on server? second? Or random (second that actually completes)?
If I understand you correctly, your question is about this flow:
client A starts uploading a huge file
client B start uploading a smaller file
client B's upload completes
client A's upload completes
Firebase's servers will handle the updates transactionally, so no partial updates will ever occur.
The last update to (completely) reach the Firebase server, is the one that will end up in the node.
If you want control over this type of concurrent access, you're probably better of using a worker queue. Of course Firebase is extremely well suited for synchronizing access to such a worker queue too. :-)
So in the flow above, no writing to your Firebase node will occur on the server until after step 3. After step 4, Firebase will write the huge file and "forgot" that the smaller file from client B ever existed. If you want to prevent such unintended overwriting, you could consider step 0 being "Client locks the upload location with a transaction call". This would essentially implement pessimistic locking on top of Firebase's optimistic locking approach.
So after the tests it seems that Firebase will perform all set operations on server sequentialy one after another, without cancelling any of them even when next one is done locally before previous completed remotely
This means that if you call set() 100 times, each set operation will affect remote data (database). For my use case this is bad because on every file change it will be entirely uploaded to firebase. I will have to write my own algorithm that will decide when to sync data with firebase.
Test consists of two scripts - one is writing data to same firebase location multiple times and second one is reading data from that location.
set_data.coffee
Firebase = require 'firebase'
firebaseUrl = 'https://my_firebase.firebaseio.com/same_location'
ref1 = new Firebase firebaseUrl
ref2 = new Firebase firebaseUrl
ref3 = new Firebase firebaseUrl
ref1.on 'value', (snapshot) -> console.log 'local same_location changed to ' + Buffer.byteLength((snapshot.val() or ''), 'utf8') + ' bytes'
data1 = ''
data2 = ''
data3 = ''
data1 += '1' for i in [1..1000*1000*2]
data2 += '2' for i in [1..1000*1000*3]
data3 += '3' for i in [1..100]
console.log 'data1 ' + Buffer.byteLength(data1, 'utf8') + ' bytes'
console.log 'data2 ' + Buffer.byteLength(data2, 'utf8') + ' bytes'
console.log 'data3 ' + Buffer.byteLength(data3, 'utf8') + ' bytes'
t1 = new Date()
ref1.set data1, (err) ->
elapsed = new Date().getTime() - t1.getTime()
console.log 'set1 finished ('+elapsed+'ms) - err is '+err
t2 = new Date()
ref2.set data2, (err) ->
elapsed = new Date().getTime() - t2.getTime()
console.log 'set2 finished ('+elapsed+'ms) - err is '+err
t3 = new Date()
ref3.set data3, (err) ->
elapsed = new Date().getTime() - t3.getTime()
console.log 'set3 finished ('+elapsed+'ms) - err is '+err
get_data.coffee
Firebase = require 'firebase'
firebaseUrl = 'https://my_firebase.firebaseio.com/same_location'
ref1 = new Firebase firebaseUrl
ref1.on 'value', (snapshot) -> console.log 'remote same_location changed to ' + Buffer.byteLength((snapshot.val() or ''), 'utf8') + ' bytes'
output from set_data.coffee
data1 2000000 bytes
data2 3000000 bytes
data3 100 bytes
local same_location changed to 2000000 bytes
local same_location changed to 3000000 bytes
local same_location changed to 100 bytes
set1 finished (118314ms) - err is null
set2 finished (149844ms) - err is null
set3 finished (149845ms) - err is null
output from get_data.coffee
remote same_location changed to 0 bytes
remote same_location changed to 2000000 bytes
remote same_location changed to 3000000 bytes
remote same_location changed to 100 bytes
I have written following code and it does not work. The following error comes while uploading file to the web services:
1.An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
2.The underlying connection was closed: An unexpected error occurred on a send.
I have used the following code for the web service and when the file size is more than 90 mb the error comes:
LocalService.IphoneService obj = new LocalService.IphoneService();
byte[] objFile = FileToByteArray(#"D:\Brijesh\My Project\WebSite5\IMG_0010.MOV");
int RtnVal = obj.AddNewProject("demo", "demo", "demo#demo.com", "demo#demo.com", 1, 2, 29, "IMG_0010.MOV", objFile,"00.00.06");
public byte[] FileToByteArray(string fileName)
{
byte[] fileContent = null;
System.IO.FileStream fs = new System.IO.FileStream(fileName, System.IO.FileMode.Open, System.IO.FileAccess.Read);
System.IO.BinaryReader binaryReader = new System.IO.BinaryReader(fs);
long byteLength = new System.IO.FileInfo(fileName).Length;
//byteLength = 94371840;
fileContent = binaryReader.ReadBytes((Int32)byteLength);
fs.Close();
fs.Dispose();
binaryReader.Close();
return fileContent;
}
No socket will transfer 200MB in one chunk. Your will receive data in chunks mostly be between 1024 and 4096 bytes(depending on your settings).
Read this data in chunks.
Reassemble your file on the server.
Then use this received file, assembled from bytes, as you need.
For an asp.net webservice:
enable webservice to receive large amounts of data
Increase the ASP.NET limits on the maximum size of SOAP messages and
the maximum number of seconds that a request is allowed to execute by
adding the configuration element to the application's
web.config file. The following code example sets the ASP.NET limit on
the maximum size of an incoming request to 400MB and the maximum
amount of time a request is allowed to execute to 5 minutes (300
seconds).
Put this in your web.config.
<configuration>
<system.web>
<httpRuntime maxMessageLength="409600"
executionTimeoutInSeconds="300"/>
</system.web>
</configuration>
Remember you will be blocking a thread for as long this request is processed. This will not scale for a large number of users.
I am trying to send a file to HTTP server via POST request (c++ and winapi), steps:
// Read file into "buff" and file size into "buffSize"
....
....
....
HINTERNET internetRoot;
HINTERNET httpSession;
HINTERNET httpRequest;
internetRoot = InternetOpen(agent_info, INTERNET_OPEN_TYPE_DIRECT, NULL, NULL, NULL);
//Connecting to the http server
httpSession = InternetConnect(internetRoot, IP,PORT_NUMBER, NULL, NULL, INTERNET_SERVICE_HTTP, NULL, NULL);
//Creating a new HTTP POST request to the default resource on the server
httpRequest = HttpOpenRequest(httpSession, TEXT("POST"), TEXT("/Post.aspx"), NULL, NULL, NULL, INTERNET_FLAG_RELOAD | INTERNET_FLAG_NO_CACHE_WRITE, NULL);
//Send POST request
HttpSendRequest(httpRequest, NULL, NULL, buff, buffSize);
//Closing handles...
In server I am recieving the file using this code (asp.net)
Stream httpStream;
try
{
httpStream = request.RequestContext.HttpContext.Request.InputStream;
}
catch (HttpException)
{
return;
}
byte[] tmp = new byte[httpStream.Length];
int bytesRead = httpStream.Read(tmp, 0, 1024 * 1024);
int totalBytesRead = bytesRead;
while (bytesRead > 0)
{
bytesRead = httpStream.Read(tmp, totalBytesRead, 1024 * 1024);
totalBytesRead += bytesRead;
}
httpStream.Close();
httpStream.Dispose();
//Save "tmp" to file...
I can send large files on local server (visual studio asp server), but I cannot send files over 1 MB to internet server. (HttpOpenRequest is failing)
is there a better way to upload files?
Caveat: My Wininet is very rusty these days.
I wonder whether you ought to be setting the "Content-Length" header yourself. Your code seems to assume that either a) you are making a HTTP/1.0 request or b) that HttpSendRequest will add the header for your (which I don't think it does).
Either way without the server being told how big the incoming request is the default configuration of IIS will reject it if it can't determine the request size itself quickly.
My guess is if you use the lpszHeaders and dwHeadersLength parameters of the HttpSendRequest function to include the appropriate "Content-Length" header the problem will be resolved.
What error do you receive? I mean what does GetLastError() returns? If you send file 800KB then it works OK? I dont really see how because HttpOpenRequest does not know about size of the data.
Maybe it timeouts? But this would mean that HttpSendRequest actually fails. It might buffer all data but since size is huge, then it takes more time than timeout allows.
use following code to query current timeouts (in ms):
InternetQueryOption(h, INTERNET_OPTION_RECEIVE_TIMEOUT, &dwReceiveTimeOut, sizeof(dwReceiveTimeOut));
InternetQueryOption(h, INTERNET_OPTION_SEND_TIMEOUT, &dwSendTimeOut, sizeof(dwSendTimeOut));
and following to set new ones:
InternetSetOption(h, INTERNET_OPTION_RECEIVE_TIMEOUT, &dwNewReceiveTimeOut, sizeof(dwNewReceiveTimeOut));
InternetSetOption(h, INTERNET_OPTION_SEND_TIMEOUT, &dwNewSendTimeOut, sizeof(dwNewSendTimeOut));