I am consuming real time data of kafka into pinot. Date time format of data is given below:
"Mon Mar 14 15:58:24 IST 2022"
I am struggling to write "dateTimeFieldSpecs" in schema file.
Can you please help me out.
Thanks,
Sukumar
you can use the transformation functions to do that for you.
First add the column to your regular schema.json. After that, add a transformation function to the table settings, usually, table.json. This function takes in the original column and re-formats it to Pinot's type.
Bare in mind though, it may be necessary to check server vs. local time zones, especially in real-time tables.
Check out these docs
Hope this helps!
Related
I have a request to alter current columns which are type of 'time' and instead of capturing just time I need to capture so called "utc time".
My idea is to create a fixed codelist with all timezones, and then to reference it to a appropriate table as FK.
My questions are:
Can column of a type 'time' hold also an information regarding time zones (utc, for example 15:00:00 +2 (gmt + 2)) and if not, could you suggest me another type for that column?
Should I maybe need to separate it into two columns? For example: [15:00:00] - StartTime, [+2:00] - UtcOffset
EF Insert: When I do inserting to the db, for that particular column, should I convert my DateTime object to for example DateTimeOffset?
Thanks in advance.
From the comments in your question, it sounds like you are building an appointment scheduling system. I'll base my answer on that, because your specific questions aren't quite aligned to the scenario you described.
First, it's important to understand that the relationship between a time zone and an offset is a one-to-many relationship. One time zone can have multiple offsets. In other words, a time zone is not an offset, but rather a time zone has an offset.
A time zone represents a geographic region where the local time is the same throughout. It is identified by a string ID, such as "America/Los_Angeles" (an IANA time zone ID) or "Pacific Standard Time" (a Windows time zone ID). In .NET, you will use them on the TimeZoneInfo object with the Id instance property or methods like FindSystemTimeZonesById.
An offset is like -07:00 or +05:30 or even +13:45. Any given offset applies only at a particular date and time. For example, in America/Los_Angeles, either -08:00 or -07:00 apply depending on whether daylight saving time is in effect at a given point in time. Keep in mind that DST is not the only reason for offsets to be different - many time zones have changed their standard time at some point in their history.
Also, it's called an offset because it is deviated from UTC by a certain amount. UTC itself always has a zero offset, delineated either by +00:00 or sometimes by Z. It's similar to GMT, except in how it is defined. UTC applies universally, everywhere. GMT technically applies only on the prime meridian. They both refer to the zero offset. You should prefer to say "UTC" in most cases.
Next, you should separate your application logic between future scheduling and present/past record keeping.
Present/past is the easier of the two. Since the moment in time has actually occurred, the local time and its offset from UTC is fixed forever. You can either store the local time and its offset in a single .NET DateTimeOffset structure (mapped to a datetimeoffset field in SQL Server). In other words, you can simply store 2021-07-27T12:00:00-08:00.
Note that you could instead store the equivalent UTC date and time, which would be 2021-07-27T20:00:00+00:00. However you've then lost the local time, and thus would need to convert back using the original time zone if you wanted to see that time. Some people prefer that, but I think it's more useful to store the original value.
For future scheduling, the situation is a bit different. Consider that the offset might not be the same for one appointment as it will be for the next appointment in the same time zone. Also consider that the definitions for which offset apply might change in between the time you schedule the appointment and when it comes around. (The likelihood of that increases the further out you schedule.)
Thus, for each location you should not store an offset, but rather a time zone identifier. Add a TimeZoneId to your object that stores each location (or each appointment depending on your model schema). Use TimeZoneInfo.GetSystemTimeZones to list the available time zones. The DisplayName property can be shown to your user, and the selected Id property gets assigned to the TimeZoneId.
Next you have to consider if you are scheduling a single appointment or creating a recurring appointment pattern.
For a single appointment you simply need the local date and time for that appointment. You can use a .NET DateTime struct (use datetime2 in SQL). Don't apply an offset, and don't convert to UTC. Just store the information provided.
For recurring appointments, you need to think through the information provided and store exactly what is given. For example, if the appointment is at 10:00 every other Tuesday, you'll need to store "10:00", "Tuesdays" and "2 week intervals". The data types for each will vary depending on how you choose to store and apply them. For example, you might use a time type for 10:00, but you could store the other values using integers. Appointments of different patterns could get stored in different ways.
Alternatively, some like to store patterns using a string containing a CRON expression. You can google that for more details.
Now you have everything you need to both schedule an appointment and record that appointment after it happens. But there's one part missing - you'll likely want some table of upcoming appointments that are easily queryable. For that, you've got a few options:
You can create a separate table via a background job of some kind. Periodically it would query all the appointments, use their information to compute the next upcoming appointment time, and insert it. You can store that in a DateTimeOffset, either as local time or as UTC. (SQL Server will always compute indexes on the equivalent UTC time either way.)
You could just add another field to your appointments that shows the next actual appointment time. You can then compute the next upcoming appointment whenever the appointment is created or updated, or when that appointment occurs, and update the table accordingly.
With either approach keep in mind that you will want to periodically check for time zone data updates (either via Windows Update or keeping the tzdata package current on Linux). You will also want to periodically re-compute future appointment times, in case that time zone data has been modified in a way that affects the appointments.
If all of this sounds super complicated - sorry, but it is. Doing scheduling worldwide across time zones right is challenging. If you want it simpler, you might want to look into a pre-made solution such as Quartz.NET, which you can integrate into your application.
I want to fetch date and time, in different timezone using SQLITE. Say, I get the value as, 20160118T010856 in input, but I require it to change to Australian, England and other timezone.
What I am currently able to do is :
select datetime('2016-01-18T04:13:39','localtime');
Which only provides me for my current local time. Kindly help.
As far as I know this is not possible in sqlite. Your best bet is to store the datetime as UTC and do the conversion outside the database.
From the documentation you could apply an offset(modifier) to a date and time, but it doesn't take Daylight Savings Time into account.
This question already has answers here:
moment.js - UTC does not work as i expect it
(2 answers)
Closed 8 years ago.
I have gone through the documentation and am a tiny bit confused about how to proceed.
There are similar questions, but none talk about parsing particular dates received in formats and swapping between local and utc dates.
I receive a local datetime, local datetime format and need to generate utc datetime from it in a particular format and this is how I think I should do it. moment(dateTime,localDateTimeFormat).utc().format(specifiedFormat);
I receive utc datetime in a particular format and have to generate locale specific datetime in a particular format. How do i do it?
moment.utc(utcDateTime, utcDateTimeFormat).toDate(); gives me javascript date i believe. How do I format it then?? Do I have to create a new moment using the generated Date object?
Another thing I could do would be getting the timezone and then formatting. I wonder if I am taking the wrong route here. Please help.
On Item 1 - Yes, that's one way to do it. However, if the output format is just going to be an ISO8601 UTC timestamp, then you can call toISOString directly on the original moment. Since UTC is implied by the output, it would be redundant to call utc() again.
On Item 2 - Just like the utc() function, there's also a local() function. Once you have a moment object, you can use toDate or format or any other of the functions described in the documentation. No, you do not need to create a new moment using the generated date object.
moment.utc(utcDateTime, utcDateTimeFormat).local().format(specifiedFormat)
Again, there's more than one way to do things here. If the utcDateTime is already in ISO8601 format, and contains either a Z or an offset like -01:00, then that will be taken into account and you can simply do this:
moment(utcDateTime).format(specifiedFormat)
On the last item you mentioned about time zones, it's difficult to tell what you are asking. You should elaborate with specific details in a new question.
We have an ASP.NET website and an SQL database hosted in US. Whenever I use the function Now() in VB.NET and getdate() in SQL, I get the US' current time. The problem is, the client is in the Philippines which is on GMT+8 Time Zone. My question is, is there any way I can set the Time Zone of a specific database and website so that when I use the functions, I'll get the Philippine's current time? How do you deal with this? As much as possible, we don't want to do subtraction or addition to the result date of the functions since in the future, clients will be from other country. It will give us headache updating the codes if we do that.
Thank you in advance!
Given that your clients may be in different time zones, you should store a timezone for the clients, that they (or you) can set as a preference for their account. Store all dates+times as UTC, and then convert to their timezone when displaying results in your interface.
This question has already been addressed to a great extent in the following question:
How to work with time zones in ASP.NET?
Follow-up:
Unfortunately, the SQL server date is a system-level setting, so it's not really something that can be manipulated on a per-session basis. It sounds like you will need to make some code changes, but you can isolate them.
Do you have a session-level variable which contains the client time zone offset? If not, create one.
Create a small date/time utility class.
In the utility class, provide 3 methods to:
(1) get the current date/time (offset to the client's time zone)
(2) pass in a database date/time to return the time offset for the client's TZ.
(3) pass in a time from the client to subtract out the client's TZ difference.
You will have to make code changes, but you can probably use those utility functions to wrap inputs and outputs everywhere, centralizing the logic. Microsoft has a page about mis-steps to avoid when using the DateTime class and manipulating time zones:
http://msdn.microsoft.com/en-us/library/ms973825.aspx#datetime_topic1a
I am working on cleaning up a bug in a large code base where no one was paying attention to local time vs. UTC time.
What we want is a way of globally ignoring time zone information on DateTime objects sent to and from our ASP.NET web services. I've got a solution for retrieve operations. Data is only returned in datasets, and I can look for DateTime columns and set the DateTimeMode to Unspecified. That solves my problem for all data passed back and forth inside a data set.
However DateTime objects are also often passed directly as parameters to the web methods. I'd like to strip off any incoming time zone information. Rather than searching through our client code and using DateTime.SpecifyKind(..) to set all DateTime vars to Undefined, I'd like to do some sort of global ASP.NET override to monitor incoming parameters and strip out the time zone information.
Is such a thing possible? Or is there another easier way to do what I want to do?
Just to reiterate -- I don't care about time zones, everyone is in the same time zone. But a couple of users have machines badly configured, wrong time zones, etc. So when they send in July 1, 2008, I'm getting June 30, 2008 22:00:00 on the server side where it's automatically converting it from their local time to the server's local time.
Update: One other possibility would be if it were possible to make a change on the client side .NET code to alter the way DateTime objects with Kind 'Undefined' are serialized.
I have dealt with this often in many applications, services, and on different platforms (.NET, Java, etc.). Please believe me that you do NOT want the long term consequences of pretending that you don't care about the time zone. After chasing lots of errors that are enormously difficult and expensive to fix, you will wish you had cared.
So, rather than stripping the time zone, you should either capture the correct time zone or force a specific time zone. If you reasonably can, get the various data sources fixed to provide a correct time zone. If they are out of your control, then force them either to the server's local time zone or to UTC.
The general industry convention is to force everything to UTC, and to set all production hardware clocks to UTC (that means servers, network devices like routers, etc.). Then you should translate to/from the user's local time zone in the UI.
If you fix it correctly now, it can be easy and cheap. If you intentionally break it further because you think that will be cheaper, then you will have no excuses later when you have to untangle the awful mess.
Note that this is similar to the common issue with Strings: there is not such thing as plain text (a String devoid of a character encoding) and there is no such thing as a plain (no time zone) time/date. Pretending otherwise is the source of much pain and heartache, and embarrassing errors.
OK, I do have a workaround for this, which depends on the fact that I only actually need the Date portion of the DateTime. I attach this property to every Date or DateTime parameter in the system
<XmlElement(DataType:="date")>
This changes the generated wsdl to have the type s:date instead of s:dateTime. (Note that simply having the type of the .NET method parameter be a Date rather than a DateTime did NOT accomplish this). So the client now only sends the date portion of the DateTime, no time info, no time zone info.
If I ever need to send a Date and Time value to the server, I'll have to use some other workaround, like making it a string parameter.
I've had issues with the time zone information as well. The problem is I'm already providing the datetime fields in UTC. Then the serialization occurs and the local offset becomes part of the date/time. The dates/times for our vendor in a different timezone were pretty messed up. I got around this problem by using the tsql convert function on the datetime fields in my select statement I used to populate my datasets. This converted the fields to a string variable, which translates nicely to a datetime value automatically on the client side. If you just want to pass the date, you can use the 101 code to provide just the date. I used 126 to provide the date and time exactly how it appears in my database columns, with the timezone information stripped out.