I am making an editor for a field with numbers. I tried a text field, but since it's a Number datatype coming in, it didn't go smoothly -- despite recasting strings as numbers etc.. it kept giving me NaN as the value. So I decided it would be best to go with a numeric stepper.
When I initially loaded it up it would drop all my decimals and only display my numbers as integers. I changed the stepIncrement to 0.1 and now it does show the decimals (a weird requirement imo).. but when I step up it occasionally gives me a value like '17.700000000000003' when I would expect 17.7. All of the numbers in my data have a single decimal place. I know I can write a dataformatter, but it seems like it shouldn't be necessary in this situation.
Is there another way I could deal with this?
You've stumbled upon the compromise of trying to represent decimal numbers in floating point binary formats like IEEE 754. Not all decimal numbers can be exactly represented. You can read up on this issue in great detail here:
http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding
You can use Number.toFixed(fractionDigits:uint) to display to an arbitrary number of decimal places.
You can use the valueFormatFunction which takes the numeric value and formats it to a string. You will need to set explicit widths on your numeric steppers to make they fit though.
in your MXML
<s:NumericStepper valueFormatFunction="stepperFormatter"/>
in your script
protected function stepperFormatter(newValue:Number):String
{
return Math.ceil(newValue).toString()
}
Related
I am reading a csv file with some really big numbers like 1327707999760, but R automatically converts it into 1.32771e+12. I've tried to assign it a double class but it didn't work because it's already a rounded value.
I've checked other posts like Preserving large numbers . People said "It's not in a "1.67E+12 format", it just won't print entirely using the defaults. R is reading it in just fine and the whole number is there." But when I tried to do some arithmetic things on them, it's just not right.
For example:
test[1,8]
[1] 1.32681e+12
test[2,8]
[1] 1.32681e+12
test[2,8]-test[1,8]
[1] 0
But I know they are different numbers!
That's not large. It is merely a representation problem. Try this:
options(digits=22)
options('digits') defaults to 7, which is why you are seeing what you do. All twelve digits are being read and stored, but not printed by default.
Excel allows custom formats: Format/Cells/Custom and enter #0
I have a spreadsheet called "MTM Lookup", this is downloaded from a specific site, in column D of this spreadsheet are values, but these values have a formula attached to them. The formula rounds these values to 0. The formulas don't round on a specific cell. They look like this =ROUND(35370.6708773751,0) or =ROUND(48368.0427161566,0). I need the values to come through with all the decimals or rounded to 10 decimals but cannot get this to happen, I can remove the formula and leave the value but it is rounded to zero. Please could anyone assist with some simple vba to either remove the =round(,0) or replace the 0 to 10 ie round(x,10).
I don't see any problem in the formula you provided.
When I put
=ROUND(35370.6708773751,0)
to a cell, I correctly see 35371 in the cell.
There are, however, two things in play here.
Formula
Cell Format
For example, when I enter the following value to the cell:
=ROUND(35370.6708773751,10)
I do see 35370.67088 as a result (after rounding to 10 places, cell format rounds it again to 5 decimal places)
I don't know why entering the value without any formula shows you 0, but this leads me to the same suspscion, i. e. that the problem is in the cell format.
You can check it by right clicking on the cell > Format Cells (in office 2010 at least) or programatically, using the following, for example:
sheets("MTM Lookup").range("A:1").numberformat = "0.0000000000"
Hope that it helps.
I have a SQLite3 table with a column having format DECIMAL(7,2), but whenever I select rows with values not having a non-zero 2nd decimal place (eg. 3.00 or 3.10), the result always has trailing zero(s) missing (eg. 3 or 3.1). Is there any way that I can apply a formatting function in the SELECT statement so that I get the required 2dp? I have tried ROUND(), but this has no effect. Otherwise I have to keep converting the resulting column values into the required format for display (using Python in my case) every time I do a SELECT statement, which is a real pain.
I don't even mind if the result is string instead of numeric, as long as it has the right number of decimal places.
Any help would be appreciated.
Alan
SQLite internally uses IEEE binary floating point arithmetic, which truly does not lend itself well to maintaining a particular number of decimals. To get that type of decimal handling would require one of:
Fixed point math, or
IEEE decimal floating point (rather uncommon), or
Handling everything as strings.
Formatting the values (converting from floating point to string) after extraction is the simplest way to implement things. You could even hide that inside some sort of wrapper so that the rest of the code doesn't have to deal with the consequences. But if you're going to do arithmetic on the value afterwards then you're better off not formatting and instead working with the value as returned by the query, because the format and reconvert back to binary floating point (which Python uses, just like the vast majority of other modern languages) loses lots of information in the reduced precision.
I need to do calculations (division or multiplication) with very large numbers. Currently I am using Double and getting the value round off problems. I can do the same calculations accurately on C# using Decimal type. I am looking for a method to do accurate calculations in VB6.0 and I couldn't find a Decimal type in VB6.0.
What is the data type used for doing arithmetic calculations with large values and without getting floating point round off problems?
Depending on your data type, you can always use Currency, which is like Decimal(19,4), or 15 digits to the left of the decimal point, and 4 to the right.
In VB6, try using the variant datatype, and casting your numbers to that using CDec, as in:
Dim myDec As Variant
myDec = CDec(1.234)
See if that works.
I have an ASP page where I have 2 variables, strActualRate and strProposed.
The values are:
strActualRate = 33.30
strProposed = 33.3
So when I write the following line to compare:
if strActualRate <> strProposed then
Response.Writr "Both are not equal!"
end if
I am getting the output "Both are not equal", even though both are the same.
I am sure that I need to use some mathematical conversion function to compare.
Can anyone tell me how to solve this ?
Thanks in advance!
If I understand correctly, you think the two values are equal but because VBScript is comparing strings rather than numbers the two are coming back as not equal.
You're correct in the conversion idea, and here's the code:
if CDbl(strActualRate) <> CDbl(strProposed) then
Response.Write "Both are not equal!"
end if
That will convert your string values to numbers to do the comparison.
Your question doesn't really add up, so I'm not really sure what the problem is. I will try to clear up some things about data types and comparison.
You are using the prefix "str" for your variables which suggests that you intend to store string values in them, however you are instead storing numeric values in them. Either you are confused about how hungarian notation is used to keep track of the data type, or the code that you posted does not look like the code that you are actually using.
The numeric value 33.30 is exactly the same as the value 33.3. If you instead would have used the string values "33.30" and "33.3", they would be two strings that are not equal.
If your code is corrected (Response.Write instead of Response.Writr) so that it runs, it will not produce any output at all. As the values are equal, the condifion in the if statement evaluates to false.
If you do in fact assign string values to the variables, the code would output "Both are not equal!". This is just as expected as the strings are not equal. If you have strings and want to compare them as numerical values, you have to comvert them:
If CDbl(strActualRate) <> CDbl(strProposed) Then
Response.Write "Both are not equal!"
End If
Try casting the values to a double in the comparison statement with CDbl()
Are you intending to perform the comparison as strings, floating point numbers or some other method? If you are comparing them as strings, then clearly they are not equal, as one of them has an extra zero on the end. If you are comparing them as floating point numbers, then you generally want to use a comparison that involves taking the difference and checking that it is smaller than some small value. This is because floating point number calculations involve some degree of inaccuracy and comparisons between them can fail because of the underlying representation.