i have to make a database access fast which contains large data of unique index mstr_nbr how can i make it fast.
in which get first mstr taking 0ms
and get next mstr takes most of the time 0ms but sometimes takes 1ms
means in 180000 for each mstr runs 12000 runs takes 1 ms which is increasing the time and if no of serial loop increases every time 12000ms second also increase this is a webspeed generated webpage how can i make it fast...anyone help
DEFINE QUERY Mstr FOR mstr scrolling.
OPEN QUERY Mstr FOR EACH mstr no-lock
where (Mstr_status = "close" or Mstr_status = "closed").
FOR EACH serial
WHERE (serial_pallet = f_pallet AND serial_f_chr11 <> "BOX")
or (serial_key begins f_pallet)
NO-LOCK BREAK BY serial_pallet by serial_parent by serial__chr11 QUERY-TUNING(LOOKAHEAD CACHE-SIZE 32768 DEBUG EXTENDED):
GET FIRST Mstr.
define variable roID as rowid no-undo.
roID = rowid(mstr).
DO WHILE NOT QUERY-OFF-END('Mstr'):
for each det fields(detnbr detmodel detlot detqty) no-lock
where (detnbr = mstr_nbr) and (detmodel = serial_parent and detlot = serial__chr11):
tinspected = tinspected + detqty.
end. /* for each */
GET NEXT Mstr.
END.
reposition mstr to rowid roID.
end.
index for mstr table
index-name field-name
badgenew 1 badgenew Active
datenew 1 datenew Active
nbridx 1 nbr Unique Primary Active
pallet 1 pallet Active
proddesc 1 proddesc Active
prodline 1 prodline Active
status 1 status Active
type 1 type Active
indexes for table serial:
actual_prod_day 1 dte04 2 serial_chr01 Active
actual_prod_line 1 serial_pallet 2 serial_dte04 3 serial_chr01 4 serial_line Active
pallet_prod 1 serial_pallet 2 serial_dte04 Active
pallet_prod_line 1 serial_pallet 2 serial_dte04 3 serial_line Active
serial_chr01 1 serial_chr01 Active
serial_chr05 1 serial_chr05 Active
serial_chr06 1 serial_chr06 Active
serial_chr11 1 serial_chr11 Active
serial_chr14 1 serial_chr14 Active
serial_dte04 1 serial_dte04 Active
serial_int01 1 serial_int01 Active
serial_line 1 serial_line Active
serial_pallet 1 serial_pallet Active
serial_parent 1 serial_parent Active
serial_serial__key 1 serial_serial__key 2 serial_parent Unique Primary Active
serial_pallet serial_key and serial_c11 all are character data type
indexes for table det:
detidx 1 detnbr 2 detpallet 3 detprodline 4 detbox 5 detlot 6 detshift Unique Primary Active
detlot 1 detlot Active
detmodel 1 detmodel Active
detnbr 1 detnbr Active
detpallet 1 detpallet Active
detprodline 1 detprodline Active
Given what we know this is how I would code it:
define temp-table tt_mstr
field mstr_nbr as integer /* or whatever the proper data type is */
index mstr_nbr-idx is primary unique /* I am assuming mstr_nbr is unique */
.
for each mstr no-lock
where mstr.mstr_status = "close"
or mstr.mstr_status = "closed":
create tt_mstr.
tt_mstr.mstr_nbr = mstr.mstr_nbr.
end.
for each serial no-lock
where ( serial_pallet = f_pallet and serial_f_chr11 <> "box" ) /* <> "box" is going to perform poorly, there may be better ways to do this */
or ( serial_key begins f_pallet ):
/* break by serial_pallet by serial_parent by serial__chr11: ** this sort of pointless, you're just adding up "tinspected", the order and the break groups have no impact */
for each det fields( detnbr detmodel detlot detqty ) no-lock
where detmodel = serial_parent and detlot = serial__chr11:
find tt_mstr where tt_mstr.mstr_nbr = detnbr no-error.
if available tt_mstr then
tinspected = tinspected + detqty.
end.
end.
Using a temp-table avoids refetching all of the "close" and "closed" records with every iteration of the "serial" table.
Maybe there is some context missing but the scrolling query and repositioning of the row seem pointless.
The selection of "serial" records does not look very efficient but I would need to know what indexes are available and what the data in serial_pallet serial_key and serial_c11 looks like. If there are just a few discrete values there may be better ways to write that.
You have single component indexes on each of the fields serial_pallet, serial_f_chr11, and serial_key. So there is not much help there.
This is speculation but if the number of discrete values in serial_f_chr11 is small you would probably be better off with a series of equality matches and OR. Suppose the valid values are BOX, JAR, BAG, and LOOSE. In that case, instead of:
where ( serial_pallet = f_pallet and serial_f_chr11 <> "box" )
or ( serial_key begins f_pallet )
you could write:
where ( serial_pallet = f_pallet and serial_f_chr11 = "jar" )
or ( serial_pallet = f_pallet and serial_f_chr11 = "bag" )
or ( serial_pallet = f_pallet and serial_f_chr11 = "loose" )
or ( serial_key begins f_pallet )
That would be even better if you have a composite index on serial_pallet + serial_f_chr11.
If the number of discrete valid serial_f_chr11 values is larger or if new values might get added then it would be better to add them to a temp-table and join on that instead.
Another option is that rather than loop over all of the master records to find matching details, select details that match the serials first. Then find the appropriate master record. That eliminates a whole layer of looping. But it depends on proper indexes in the "det" table. You must have an index that has serial_parent and detlot as leading components.
If mstr_nbr is not the same field as "nbr" (as shown in your index listing) then you need to build the TT and add an index on mstr_nbr. If, however, nbr is actually the same as mstr_nbr then you could skip the TT and directly query the database table efficiently.
Related
I am new to progress 4gl and below is the query used to add all fields from a table to dynamic temp table except few fields but I am not sure how to add only required fields to dynamic temp table. Please help to modify the query I shared.
/* p-ttdyn2.p - a join of 2 tables */
DEFINE VARIABLE tth4 AS HANDLE.
DEFINE VARIABLE btth4 AS HANDLE.
DEFINE VARIABLE qh4 AS HANDLE.
DEFINE VARIABLE bCust AS HANDLE.
DEFINE VARIABLE bOrder AS HANDLE.
DEFINE VARIABLE i AS INTEGER.
DEFINE VARIABLE fldh AS HANDLE EXTENT 15.
bCust = BUFFER customer:HANDLE.
bOrder = BUFFER order:HANDLE.
CREATE TEMP-TABLE tth4.
tth4:ADD-FIELDS-FROM(bCust,"address,address2,phone,city,comments").
tth4:ADD-FIELDS-FROM(bOrder,"cust-num,carrier,instructions,PO,terms").
tth4:TEMP-TABLE-PREPARE("CustOrdJoinTT").
btth4 = tth4:DEFAULT-BUFFER-HANDLE.
FOR EACH customer WHERE cust.cust-num < 6, EACH order OF customer:
btth4:BUFFER-CREATE.
btth4:BUFFER-COPY(bCust).
btth4:BUFFER-COPY(bOrder).
END.
/* Create Query */
CREATE QUERY qh4.
qh4:SET-BUFFERS(btth4).
qh4:QUERY-PREPARE("for each CustOrdJoinTT").
qh4:QUERY-OPEN.
REPEAT WITH FRAME zz DOWN:
qh4:GET-NEXT.
IF qh4:QUERY-OFF-END THEN LEAVE.
REPEAT i = 1 TO 15:
fldh[i] = btth4:BUFFER-FIELD(i).
DISPLAY fldh[i]:NAME FORMAT "x(15)"
fldh[i]:BUFFER-VALUE FORMAT "x(20)".
END.
END.
btth4:BUFFER-RELEASE.
DELETE OBJECT tth4.
DELETE OBJECT qh4.
ADD-FIELDS-FROM only supports excluding fields that are not needed. Instead you can use ADD-LIKE-FIELD multiple times:
CREATE TEMP-TABLE tth4.
tth4:ADD-LIKE-FIELD("address", "customer.address").
tth4:ADD-LIKE-FIELD("address2", "customer.address2").
tth4:ADD-LIKE-FIELD("phone", customer.phone").
...
tth4:ADD-LIKE-FIELD("cust-num", "Order.cust-num").
...
tth4:TEMP-TABLE-PREPARE("CustOrdJoinTT").
btth4 = tth4:DEFAULT-BUFFER-HANDLE.
Depending on your use case, you can also invert the required field list to an except field list:
var handle ht,hb.
var longchar lcjson.
function invertFields returns character (
i_hb as handle,
i_crequired as char
):
var char cexcept,cfield.
var int ic.
do ic = 1 to i_hb:num-fields:
cfield = i_hb:buffer-field( ic ):name.
if lookup( cfield, i_crequired ) = 0 then
cexcept = cexcept + ',' + cfield.
end.
return substring( cexcept, 2 ).
end function.
create temp-table ht.
ht:add-fields-from(
buffer customer:handle,
invertFields( buffer customer:handle, "CustNum,Name" )
).
ht:temp-table-prepare( 'tt' ).
hb = ht:default-buffer-handle.
hb:buffer-create().
assign
hb::CustNum = 1
hb::Name = 'test'
.
hb:write-json( 'longchar', lcjson, true ).
message string( lcjson ).
https://abldojo.services.progress.com/?shareId=624993253fb02369b25437c4
I have the query
select d.did, count ( h.did ), unique_interested
from dealer as d
left outer join house as h
on h.did = d.did
left outer join (
-- cid = customer id
select hid, count (cid) as unique_interested
from is_interested
group by hid
) as ok
on h.hid = ok.hid
group by d.did
order by d.did asc
;
which is supposed to select the number of houses that each dealer is dealing, and the number of unique customers interested in said houses (as in the number of customers per dealer). This should happen even if the dealers have no houses to deal at the moment, which is why I'm using left outer joins when constructing the table the columns will be picked from.
Now, running this query against my database produces the following output:
d.did count ( h.did) unique_interested
----- -------------- ----------------
1 3
2 3 1
3 0
As you can see, instead of printing 0 in the last column, count returns null, when there is a null in one of the aparments produced by the last part of the join (as in cid is null):
select hid, count ( cid ) as unique_interested
from is_interested
group by hid
I know this is because there are apartments in the table produced by from, that no-one is interested in. But shouldn't count produce 0 instead of the actual column value null in every case?
Any explanation as to why this is happening would be appreciated, as it would lead me towards an answer to another question, which is "Why am I not getting the right number of unique interested customers per dealer from the table is_interested?", as with the current state of my database, the output should look more like:
d.did count ( h.did) unique_interested
----- -------------- ----------------
1 3 2
2 3 2
3 0 0
Trying to optimize a query, which has multiple counts for objects in subordinate table (used aliases in SQLAlchemy). In Witch Academia terms, something like this:
SELECT
exam.id AS exam_id,
exam.name AS exam_name,
count(tried_witch.id) AS tried,
count(passed_witch.id) AS passed,
count(failed_witch.id) AS failed
FROM exam
LEFT OUTER JOIN witch AS tried_witch
ON tried_witch.exam_id = exam.id AND
tried_witch.is_failed = 0 AND
tried_witch.status != "passed"
LEFT OUTER JOIN witch AS passed_witch
ON passed_witch.exam_id = exam.id AND
passed_witch.is_failed = 0 AND
passed_witch.status = "passed"
LEFT OUTER JOIN witch AS failed_witch
ON failed_witch.exam_id = exam.id AND
failed_witch.is_failed = 1
GROUP BY exam.id, exam.name
ORDER BY tried ASC
LIMIT 20
Number of witches can be large (hundreds of thousands), number of exams is lower (hundreds), so the above query is quite slow. In a lot of similar questions I've found answers, which propose the above, but I feel like a totally different approach is needed here. I am stuck at coming up with alternative. NB, there is a need to order by calculated counts. It is also important to have zeros as counts, of course, where due. (do not pay attention to a somewhat funny model: witches can easily clone themselves to go to multiple exams, thus per exam identity)
With one EXISTS subquery, which is not reflected in the above and does not influence the ouotcome, the situation is:
# Query_time: 1.135747 Lock_time: 0.000209 Rows_sent: 20 Rows_examined: 98174
# Rows_affected: 0
# Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes
# Filesort: Yes Filesort_on_disk: No Merge_passes: 0 Priority_queue: No
Updated query, which is still quite slow:
SELECT
exam.id AS exam_id,
exam.name AS exam_name,
count(CASE WHEN (witch.status != "passed" AND witch.is_failed = 0)
THEN witch.id
ELSE NULL END) AS tried,
count(CASE WHEN (witch.status = "passed" AND witch.is_failed = 0)
THEN witch.id
ELSE NULL END) AS passed,
count(CASE WHEN (witch.is_failed = 1)
THEN witch.id
ELSE NULL END) AS failed
FROM exam
LEFT OUTER JOIN witch ON witch.exam_id = exam.id
GROUP BY exam.id, exam.name
ORDER BY tried ASC
LIMIT 20
Indexing is the key to get performance of the query.
I do not know MariaDB at all, so not sure what the possibilities are. But if it is anything like Microsoft SQL Server, then here is what I would try:
Create ONE composite index covering ALL the required columns: witch_id, status and is_failed. If the query uses that index, that should be it. Here the order of the included columns might be very important. Then profile the query in order to understand if the index is used. See Optimization and Indexes documentation page.
Consider Generated (Virtual and Persistent) Columns.
It looks like all the information for classification of the witch into tried, passed or failed bucket is contained in the row for witch. Therefore, you can basically create those virtual columns on the database table directly and use PERSISTENT option. This option allows creating index on it. Then you can create an index specifically for this query containing witch_id and three virtual columns: tried, passed and failed. Make sure you query uses it, and that should be pretty good. The query will then look very simple:
SELECT exam.id,
exam.name,
sum(witch.tried) AS tried,
sum(witch.passed) AS passed,
sum(witch.failed) AS failed
FROM exam
INNER JOIN witch ON exam.id = witch.exam_id
GROUP BY exam.id,
exam.name
ORDER BY sum(witch.tried)
LIMIT 20
Although query simple comparisons and AND/OR clauses, you are basically offloading the calculation of the 3 statuses to the database during INSERT/UPDATE. Then during SELECT you query should be much faster.
Your example does not specify any result filtering (WHERE clause), but if you have one, it might also have an impact on the way one optimises indices for query performance.
Original answer: Below is the originally proposed change to the query.
Here i assume that indexing part of the optimisation has been already done.
Could you try with SUM instead of COUNT?
SELECT exam.id,
exam.name,
sum(CASE
WHEN (witch.is_failed = 0
AND witch.status != 'passed') THEN 1
ELSE 0
END) AS tried,
sum(CASE
WHEN (witch.is_failed = 0
AND witch.status = 'passed') THEN 1
ELSE 0
END) AS passed,
sum(CASE
WHEN (witch.is_failed = 1) THEN 1
ELSE 0
END) AS failed
FROM exam
INNER JOIN witch ON exam.id = witch.exam_id
GROUP BY exam.id,
exam.name
ORDER BY sum(CASE
WHEN (witch.is_failed = 0
AND witch.status != 'passed') THEN 1
ELSE 0
END)
LIMIT 20
The rest:
Given you have specified sqlalchemy in your answer, here is the sqlalchemy code, which i used to model and generate the query:
# model
class Exam(Base):
id = Column(Integer, primary_key=True)
name = Column(String)
class Witch(Base):
id = Column(Integer, primary_key=True)
exam_id = Column(Integer, ForeignKey('exam.id'))
is_failed = Column(Integer)
status = Column(String)
exam = relationship(Exam, backref='witches')
# computed fields
#hybrid_property
def tried(self):
return self.is_failed == 0 and self.status != 'passed'
#hybrid_property
def passed(self):
return self.is_failed == 0 and self.status == 'passed'
#hybrid_property
def failed(self):
return self.is_failed == 1
# computed fields: expression
#tried.expression
def _tried_expression(cls):
return case([(and_(
cls.is_failed == 0,
cls.status != 'passed',
), 1)], else_=0)
#passed.expression
def _passed_expression(cls):
return case([(and_(
cls.status == 'passed',
cls.is_failed == 0,
), 1)], else_=0)
#failed.expression
def _failed_expression(cls):
return case([(cls.is_failed == 1, 1)], else_=0)
and:
# query
q = (
session.query(
Exam.id, Exam.name,
func.sum(Witch.tried).label("tried"),
func.sum(Witch.passed).label("passed"),
func.sum(Witch.failed).label("failed"),
)
.join(Witch)
.group_by(Exam.id, Exam.name)
.order_by(func.sum(Witch.tried))
.limit(20)
)
I'm new to peoplesoft. I need a help in understanding the rowset and I have a requirement where i have 3 levels.
On level 1 i have a checkbox and when I open a component the value of the checkbox on level 1 should be passed and display to the level 2 grid for all rows.
For example
level0 - record1
level1 - record2 (Scroll Area)
level2 - record3 (grid)
When i access the page it should have values like this
Record2.field1 = Y => Row1 Record3.field1 = Y
Row2 Record3.field1 = Y
Record2.field1 = N => Row1 Record3.field1 = N
I have written the code at level2 record.field rowinit peoplecode event. but the problem is the same record field is used in level 0 as well. Is there a way where I can avoid using for loop as there could be n number of rows in the grid which might create a performance issue during page opening.
Thanks in advance,
Rowinit will fire for each rows in the scroll. So if you have a loop in the rowinit, loop will execute for each row.
If you want the check box to be set only during the component load, you can add the peoplecode in Component PostBuild.
&rsLevel1 = GetLevel0()(1).GetRowSet(Scroll.Level1);
for &nCnt1 = 1 to &rsLevel1.activerowcount
&rsLevel2 = &rsLevel1(&nCnt1).GetRowset(Scroll.Level2);
for &nCnt2 = 1 to &rsLevel2.activerowcount
&rsLevel2(&nCnt2).Level2.Check_box.value = &rsLevel1(&nCnt1).Level1.Check_box.value
end-for;
end-for;
I'm trying to improve a transit scheduling table by adding a column and flagging some rows to indicate they are the last stop for each trip.
Each trip will have many rows showing its stops and their sequence along the trip. I want to update the LastStop column with a '1' if the Sequence number is the highest for that trip.
I think the following SQL is on the right track but I am getting a "no such column: s1.stop_sequence" so I have no idea if I'm even on the right track until this unobvious to me error is resolved. I am a SQL lightweight barely beyond novice level. Stop_Sequence is definitely the correct name for the column.
UPDATE stop_times
SET LastStop = '1'
WHERE stop_sequence =(
SELECT max(st.stop_sequence)
FROM stop_times s1
WHERE s1.trip_id = trip_id
)
AND
trip_id = s1.trip_id
AND
stop_ID = s1.stop_id;
A simplified version of sample data is below.
TripID Stop Sequence LastStop
665381 1766 1
665381 3037 2
665381 3038 3 1
667475 1130 1
667475 2504 2 1
644501 2545 1
644501 3068 2
644501 2754 3
644501 3069 4
644501 2755 5 1
You cannot refer to a column in the subquery from the outer query.
Furthermore, the filter trip_id = s1.trip_id is duplicated, and you do not want to filter on stop_id because that would prevent the MAX from looking at any other stops of the trip.
Try this:
UPDATE stop_times
SET LastStop = '1'
WHERE Stop_Sequence = (SELECT MAX(Stop_Sequence)
FROM stop_times s1
WHERE s1.Trip_ID = stop_times.Trip_ID)
Alternatively, a last stop is a stop for which no other stop with a larger sequence number in the same trip exists:
UPDATE stop_times
SET LastStop = '1'
WHERE NOT EXISTS (SELECT 1
FROM Stop_Sequence s1
WHERE s1.Trip_ID = stop_times.Trip_ID
AND s1.Stop_Sequence > stop_times.Stop_Sequence)
This will work for you, as long as stops field is always less than 1000 (use bigger multiplier if it is):
UPDATE stop_times
SET laststop = 1
WHERE tripid*1000+sequence IN (
SELECT tripid*1000+sequence FROM (
SELECT tripid, max(sequence) AS sequence
FROM stop_times
GROUP BY 1
)
)
I would have written this using tuple syntax, but SQLite does not support it:
UPDATE stop_times
SET laststop = 1
WHERE (tripid, sequence) IN (
SELECT (tripid, sequence) FROM (
SELECT tripid, max(sequence) AS sequence
FROM stop_times
GROUP BY 1
)
)
Sorry, no SQLFiddle - it does not seem to work for me today.