I need to get the exact input value of double spin box. I set 8 decimal digit for my Spinbox. When I type 0,6000000 what I get in the debug is 0,59999999999998 as in the picture (sometimes it is 0,6000000000002).
I tried to convert this value to 0,60000000 using QString::number and convert it again to double. What I receive is the old value again (0,59999999999998).
But when I print these values out, it prints 0,60000000.
Because I have my calculation after that, so I need the exact input value, otherwise my calculation will be wrong. In this case I should become exact 0,60000000). How can I do that?
Related
I am a somewhat newbie to SQLite (and KMyMoney). KMyMoney (an open source personal finance manager) allows one-click exporting data into an SQLite database.
On browsing the SQLite database output, the dollar amount data is stored in a table called kmmSplits as several text fields in a strange format based on “value” and “valueFormatted” (see screen shot below). The “value” field is apparently written as a division equation (in a text format) which apparently yields the “valueFormatted” field (again in text format). The “valueFormatted is the correct number amount but the problem is that parenthesis are used to indicate a negative number instead of a simple minus in front of the value. This is apparently an accounting number format, but I don’t know how to parse this into a float value for running calculated SQL queries, etc. The positive values (without parenthesis) are no problem to convert to FLOATS.
I’ve tried using the CAST to FLOAT function but this does not do the division math, nor does it convert parenthesis into negative values (see screen shot).
The basic question is: how to parse a text value containing parenthesis in the “valueFormatted field (accounting money format) into a common number format OR, alternatively, how to convert a division equation in the “value” field to an actual calculation.
Use a CASE expression to check if valueFormatted is a numeric value inside parentheses and if it is multiply -1 with the substring starting from the 2nd char (the closing parenthesis will be discarded by SQLite during this implicit type casting):
SELECT *,
CASE
WHEN valueFormatted LIKE '(%)' THEN (-1) * SUBSTR(valueFormatted, 2)
ELSE valueFormatted
END AS value
FROM kmmSQLite;
Or, replace '(' with ''-'' and add 0 to covert the result to a number:
SELECT *,
REPLACE(valueFormatted, '(', '-') + 0 AS value
FROM kmmSQLite;
I have an exponential value for e.g. 3.22122E+23
In Marklogic when I try- xs:decimal(3.22122E+23)
I get this error:
[1.0-ml] XDMP-CAST: (err:FORG0001) xs:decimal(xs:double("3.22122E23")) -- Invalid cast: xs:double("3.22122E23") cast as xs:decimal
A lower value for e.g. xs:decimal(3.22122E+18) gives me the correct result i.e. 3221220000000000000.
I see that this is because of decimal overflow and cannot be represented as a decimal data type but is there any way in Marklogic to handle and calculate such huge values?
Same question applies for the negative values(3.22122E-23) where I can handle and display data above 20 decimal places.
It would be helpful to clarify what kind of logic or calculations you are trying to accomplish and why exactly you need to convert the value to decimal. For example, to "display" the double value, you can use the standard format-number function without any conversion to decimal:
let $x := xs:double(3.22122E+23)
return format-number($x,"#,##0.00")
yields:
322,122,000,000,000,000,000,000.00
See https://docs.marklogic.com/fn:format-number for details regarding fn:format-number() usage.
See https://help.marklogic.com/Knowledgebase/Article/View/487/0/marklogic-server-and-the-decimal-type-implementation for details of the limitations of the xs:decimal type.
I have a table with a numeric(6,4) column (x.xxxx)
In the select statement, I have to do math on this column which results in the value being typed as a REAL. I need to display the results using numeric(6,4) though and can't seem to see how to do this.
I tried "cast(column as numeric(6,4))" which works in other databases, but the displayed value is still REAL format with more than 4 digits to the right of the decimal. I need to get it back to the "x.xxxx" format from REAL.
I have a spreadsheet called "MTM Lookup", this is downloaded from a specific site, in column D of this spreadsheet are values, but these values have a formula attached to them. The formula rounds these values to 0. The formulas don't round on a specific cell. They look like this =ROUND(35370.6708773751,0) or =ROUND(48368.0427161566,0). I need the values to come through with all the decimals or rounded to 10 decimals but cannot get this to happen, I can remove the formula and leave the value but it is rounded to zero. Please could anyone assist with some simple vba to either remove the =round(,0) or replace the 0 to 10 ie round(x,10).
I don't see any problem in the formula you provided.
When I put
=ROUND(35370.6708773751,0)
to a cell, I correctly see 35371 in the cell.
There are, however, two things in play here.
Formula
Cell Format
For example, when I enter the following value to the cell:
=ROUND(35370.6708773751,10)
I do see 35370.67088 as a result (after rounding to 10 places, cell format rounds it again to 5 decimal places)
I don't know why entering the value without any formula shows you 0, but this leads me to the same suspscion, i. e. that the problem is in the cell format.
You can check it by right clicking on the cell > Format Cells (in office 2010 at least) or programatically, using the following, for example:
sheets("MTM Lookup").range("A:1").numberformat = "0.0000000000"
Hope that it helps.
I am making an editor for a field with numbers. I tried a text field, but since it's a Number datatype coming in, it didn't go smoothly -- despite recasting strings as numbers etc.. it kept giving me NaN as the value. So I decided it would be best to go with a numeric stepper.
When I initially loaded it up it would drop all my decimals and only display my numbers as integers. I changed the stepIncrement to 0.1 and now it does show the decimals (a weird requirement imo).. but when I step up it occasionally gives me a value like '17.700000000000003' when I would expect 17.7. All of the numbers in my data have a single decimal place. I know I can write a dataformatter, but it seems like it shouldn't be necessary in this situation.
Is there another way I could deal with this?
You've stumbled upon the compromise of trying to represent decimal numbers in floating point binary formats like IEEE 754. Not all decimal numbers can be exactly represented. You can read up on this issue in great detail here:
http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding
You can use Number.toFixed(fractionDigits:uint) to display to an arbitrary number of decimal places.
You can use the valueFormatFunction which takes the numeric value and formats it to a string. You will need to set explicit widths on your numeric steppers to make they fit though.
in your MXML
<s:NumericStepper valueFormatFunction="stepperFormatter"/>
in your script
protected function stepperFormatter(newValue:Number):String
{
return Math.ceil(newValue).toString()
}