do you know how to convert double to long in kql without rounding down? i used tolong function but its rounding down i.e 0.38 -> 0. I'm looking for a way to return the same value and convert it to long.
The long data type represents a signed, 64-bit wide, integer.
So 0.38 is simply not a long, and if you try to do the conversion, the value will be rounded down to the nearest integer value.
See more details in the doc.
Related
I need to get the exact input value of double spin box. I set 8 decimal digit for my Spinbox. When I type 0,6000000 what I get in the debug is 0,59999999999998 as in the picture (sometimes it is 0,6000000000002).
I tried to convert this value to 0,60000000 using QString::number and convert it again to double. What I receive is the old value again (0,59999999999998).
But when I print these values out, it prints 0,60000000.
Because I have my calculation after that, so I need the exact input value, otherwise my calculation will be wrong. In this case I should become exact 0,60000000). How can I do that?
In my data set there is a field that is currently a character field and I need to convert it to a numeric one. the problem is not only are there '%' signs hard coded in the data but there are decimal points in there as well and places after the decimal points is not consistent. AKA...
42.01%
8.1%
22%
.05%
I substringed off the % sign but is there a way to just cut the decimal point off and everything after it so then I can just cast it as an integer?
thanks all
Cut % then convert to double. Then apply ceiling function.
What I had to do was put leading 0's on the front of the element because cognos was not reading the entries that started with decimal places. then I had to substring the % off and trim and then cast as a number.
I pull data from a websource and it returns time in the format 'YYYY-MM-DDTHH:MM:SS.SSSZ'. Since I would be doing a lot of comparisons on this column, I thought it would be best to store them as the number of millliseconds since epoch (I need millisecond precision!!). What would be the best way to convert the incoming datetime string to number of milliseconds?
I tried to use
strftime('%s', datetimeString) - gives back the number of seconds since epoch till datetimeString
strftime('%f', datetimeString) - gives back only the SS.SSS part!!
I'm guessing it has something to do with SQLite not having actual Date or Datetime types (section 1.2). Instead it formats them as one of:
an ISO8601 string (TEXT type)
a floating point number of days (REAL type)
an integer number of seconds since the epoch time (INTEGER type)
The REAL data type may not have enough precision to store a date, time, and fractional seconds in a single field.
It seems the answer is one of:
store them as two separate fields, the datetime in one, and the fractional seconds in another
do the integer number of milliseconds calculation yourself and store the integer result in a single column, so long as the maximum level of fractional second precision you need will fit in a signed integer of 8-bytes or fewer (the maximum size for the INTEGER type in SQLite3)
switch databases to a DBMS that supports real date/datetime types
Without switching your DB, and if you never need sub-millisecond precision, then I'd at least try the "manual calculation + single integer column" solution and see if it works.
I am making an editor for a field with numbers. I tried a text field, but since it's a Number datatype coming in, it didn't go smoothly -- despite recasting strings as numbers etc.. it kept giving me NaN as the value. So I decided it would be best to go with a numeric stepper.
When I initially loaded it up it would drop all my decimals and only display my numbers as integers. I changed the stepIncrement to 0.1 and now it does show the decimals (a weird requirement imo).. but when I step up it occasionally gives me a value like '17.700000000000003' when I would expect 17.7. All of the numbers in my data have a single decimal place. I know I can write a dataformatter, but it seems like it shouldn't be necessary in this situation.
Is there another way I could deal with this?
You've stumbled upon the compromise of trying to represent decimal numbers in floating point binary formats like IEEE 754. Not all decimal numbers can be exactly represented. You can read up on this issue in great detail here:
http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding
You can use Number.toFixed(fractionDigits:uint) to display to an arbitrary number of decimal places.
You can use the valueFormatFunction which takes the numeric value and formats it to a string. You will need to set explicit widths on your numeric steppers to make they fit though.
in your MXML
<s:NumericStepper valueFormatFunction="stepperFormatter"/>
in your script
protected function stepperFormatter(newValue:Number):String
{
return Math.ceil(newValue).toString()
}
I need to do calculations (division or multiplication) with very large numbers. Currently I am using Double and getting the value round off problems. I can do the same calculations accurately on C# using Decimal type. I am looking for a method to do accurate calculations in VB6.0 and I couldn't find a Decimal type in VB6.0.
What is the data type used for doing arithmetic calculations with large values and without getting floating point round off problems?
Depending on your data type, you can always use Currency, which is like Decimal(19,4), or 15 digits to the left of the decimal point, and 4 to the right.
In VB6, try using the variant datatype, and casting your numbers to that using CDec, as in:
Dim myDec As Variant
myDec = CDec(1.234)
See if that works.